Project One - Face Mask Detection

  • Domain: Entertainment
  • Context: Company X owns a movie application and repository which caters movie streaming to millions of users on subscription basis. Company wants to automate the process of cast and crew information in each scene from a movie such that when a user pauses on the movie and clicks on cast information button, the app will show details of the actor in the scene. Company has an in-house computer vision and multimedia experts who need to detect faces from screen shots from the movie scene.
  • Data Description: The dataset comprises of images and its mask where there is a human face. -Project Objective: Face detection from training images

NOTE: The images might not get rendered in html format. For that please check the notebook.

Import Libraries

In [ ]:
import numpy as np # mathematical manipulations
import pandas as pd # data manipulations

# for visualisation
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns
import cv2

# splitting into train and test sets
from sklearn.model_selection import train_test_split

# Neural Network libraries
import tensorflow as tf

# ignore warnings
import warnings
warnings.filterwarnings("ignore")

# initialize random number generator
import random
# set the seed for consistent results for multiple runs
seed = 7
np.random.seed(seed)

Data Warehouse

In [ ]:
# copy data to current working directory
!cp '/content/drive/MyDrive/Colab Notebooks/images.npy' .
In [ ]:
# verify data copy
!ls -l
total 1325080
drwx------ 5 root root       4096 Mar 20 12:43 drive
-rw------- 1 root root 1356868279 Mar 20 12:44 images.npy
drwxr-xr-x 1 root root       4096 Mar  9 14:48 sample_data
In [ ]:
# load dataset as NumPy array
dataset = np.load(file='/content/images.npy',allow_pickle=True)
In [ ]:
# verify type of dataset
type(dataset)
Out[ ]:
numpy.ndarray
In [ ]:
# verify shape of dataset
dataset.shape
Out[ ]:
(409, 2)
In [ ]:
# observe the first record
dataset[0]
Out[ ]:
array([array([[[42, 37, 34],
               [56, 51, 48],
               [71, 66, 63],
               ...,
               [23, 33, 34],
               [26, 36, 37],
               [28, 38, 39]],

              [[40, 35, 32],
               [51, 46, 43],
               [64, 59, 56],
               ...,
               [27, 36, 35],
               [24, 33, 32],
               [26, 35, 34]],

              [[43, 38, 35],
               [51, 46, 43],
               [61, 56, 53],
               ...,
               [28, 30, 27],
               [33, 35, 32],
               [35, 37, 34]],

              ...,

              [[56, 47, 40],
               [57, 48, 41],
               [61, 52, 45],
               ...,
               [67, 48, 42],
               [55, 35, 28],
               [60, 40, 33]],

              [[53, 44, 37],
               [54, 45, 38],
               [57, 48, 41],
               ...,
               [59, 40, 34],
               [60, 40, 33],
               [54, 34, 27]],

              [[53, 44, 37],
               [54, 45, 38],
               [57, 48, 41],
               ...,
               [59, 40, 34],
               [70, 50, 43],
               [64, 44, 37]]], dtype=uint8),
       list([{'label': ['Face'], 'notes': '', 'points': [{'x': 0.08615384615384615, 'y': 0.3063063063063063}, {'x': 0.1723076923076923, 'y': 0.45345345345345345}], 'imageWidth': 650, 'imageHeight': 333}, {'label': ['Face'], 'notes': '', 'points': [{'x': 0.583076923076923, 'y': 0.2912912912912913}, {'x': 0.6584615384615384, 'y': 0.46846846846846846}], 'imageWidth': 650, 'imageHeight': 333}])],
      dtype=object)
In [ ]:
# seeing image part
dataset[0,0]
Out[ ]:
array([[[42, 37, 34],
        [56, 51, 48],
        [71, 66, 63],
        ...,
        [23, 33, 34],
        [26, 36, 37],
        [28, 38, 39]],

       [[40, 35, 32],
        [51, 46, 43],
        [64, 59, 56],
        ...,
        [27, 36, 35],
        [24, 33, 32],
        [26, 35, 34]],

       [[43, 38, 35],
        [51, 46, 43],
        [61, 56, 53],
        ...,
        [28, 30, 27],
        [33, 35, 32],
        [35, 37, 34]],

       ...,

       [[56, 47, 40],
        [57, 48, 41],
        [61, 52, 45],
        ...,
        [67, 48, 42],
        [55, 35, 28],
        [60, 40, 33]],

       [[53, 44, 37],
        [54, 45, 38],
        [57, 48, 41],
        ...,
        [59, 40, 34],
        [60, 40, 33],
        [54, 34, 27]],

       [[53, 44, 37],
        [54, 45, 38],
        [57, 48, 41],
        ...,
        [59, 40, 34],
        [70, 50, 43],
        [64, 44, 37]]], dtype=uint8)
In [ ]:
# observing mask, label and bounding box
dataset[0,1]
Out[ ]:
[{'imageHeight': 333,
  'imageWidth': 650,
  'label': ['Face'],
  'notes': '',
  'points': [{'x': 0.08615384615384615, 'y': 0.3063063063063063},
   {'x': 0.1723076923076923, 'y': 0.45345345345345345}]},
 {'imageHeight': 333,
  'imageWidth': 650,
  'label': ['Face'],
  'notes': '',
  'points': [{'x': 0.583076923076923, 'y': 0.2912912912912913},
   {'x': 0.6584615384615384, 'y': 0.46846846846846846}]}]
In [ ]:
# separating data into images and masks
images = dataset[:,0]
masks = dataset[:,1]
In [ ]:
# verify images shape
images.shape
Out[ ]:
(409,)
In [ ]:
# verify masks shape
masks.shape
Out[ ]:
(409,)

Observations:

  • The data is provided as a numpy array.
  • We have 409 images for training a face detection model.
  • The dataset comprises of an actual image as an array and the associated labels, mask and bounding box co-ordinates.
  • The bounding box co-ordinates are in scaled form.

Visualisation

In [ ]:
plt.imshow(images[408])
Out[ ]:
<matplotlib.image.AxesImage at 0x7f7a00bb9f10>
In [ ]:
display(masks[408])
[{'imageHeight': 493,
  'imageWidth': 656,
  'label': ['Face'],
  'notes': '',
  'points': [{'x': 0.3201219512195122, 'y': 0.2839756592292089},
   {'x': 0.4009146341463415, 'y': 0.4198782961460446}]},
 {'imageHeight': 493,
  'imageWidth': 656,
  'label': ['Face'],
  'notes': '',
  'points': [{'x': 0.4557926829268293, 'y': 0.38742393509127787},
   {'x': 0.5442073170731707, 'y': 0.5618661257606491}]},
 {'imageHeight': 493,
  'imageWidth': 656,
  'label': ['Face'],
  'notes': '',
  'points': [{'x': 0.6707317073170732, 'y': 0.38336713995943206},
   {'x': 0.7134146341463414, 'y': 0.4746450304259635}]},
 {'imageHeight': 493,
  'imageWidth': 656,
  'label': ['Face'],
  'notes': '',
  'points': [{'x': 0.16615853658536586, 'y': 0.4665314401622718},
   {'x': 0.22560975609756098, 'y': 0.563894523326572}]},
 {'imageHeight': 493,
  'imageWidth': 656,
  'label': ['Face'],
  'notes': '',
  'points': [{'x': 0.07164634146341463, 'y': 0.539553752535497},
   {'x': 0.11280487804878049, 'y': 0.5862068965517241}]},
 {'imageHeight': 493,
  'imageWidth': 656,
  'label': ['Face'],
  'notes': '',
  'points': [{'x': 0.25, 'y': 0.24746450304259635},
   {'x': 0.2850609756097561, 'y': 0.30425963488843816}]},
 {'imageHeight': 493,
  'imageWidth': 656,
  'label': ['Face'],
  'notes': '',
  'points': [{'x': 0.4847560975609756, 'y': 0.2332657200811359},
   {'x': 0.5274390243902439, 'y': 0.3225152129817444}]},
 {'imageHeight': 493,
  'imageWidth': 656,
  'label': ['Face'],
  'notes': '',
  'points': [{'x': 0.5929878048780488, 'y': 0.31643002028397565},
   {'x': 0.6204268292682927, 'y': 0.359026369168357}]},
 {'imageHeight': 493,
  'imageWidth': 656,
  'label': ['Face'],
  'notes': '',
  'points': [{'x': 0.7728658536585366, 'y': 0.41379310344827586},
   {'x': 0.801829268292683, 'y': 0.45841784989858014}]},
 {'imageHeight': 493,
  'imageWidth': 656,
  'label': ['Face'],
  'notes': '',
  'points': [{'x': 0.8033536585365854, 'y': 0.5152129817444219},
   {'x': 0.850609756097561, 'y': 0.6267748478701826}]}]
In [ ]:
# display random five images with bounding boxes - these would actually correspond to the  mask locations
for i in range(0,5):
  #Pickup a random image number
  img_num = np.random.randint(0, images.shape[0])

  #Read the image
  img = images[img_num]

  #Draw rectangle(s) as per bounding box information
  for dictionary in masks[img_num]:
    # get image height and width for scaling
    image_height = dictionary['imageHeight']
    image_width = dictionary['imageWidth']

    # get min box coordinates
    xmin = np.float32(dictionary['points'][0]['x'] * image_width)
    ymin = np.float32(dictionary['points'][0]['y'] * image_height)

    # get max box coordinates
    xmax = np.float32(dictionary['points'][1]['x'] * image_width)
    ymax = np.float32(dictionary['points'][1]['y'] * image_height)

    #Get Label
    label = dictionary['label'][0]

    #Add bounding box
    cv2.rectangle(img, (xmin,ymin), (xmax, ymax), (0,255,0), 2)

  #Draw image using matplotlib
  plt.figure(figsize=(10,7))
  plt.imshow(img)
  plt.show()

Observations:

  • Images are of varying height and width. So, resize is required.
  • The bounding boxes or the masks are in form of rectangles.
  • The masks are scaled versions of actuals.
  • The task seems to revolve around detection or segmentation.
  • For the notebook we will focus on segmentation approach for face mask detection.

Data Preparation

In [ ]:
# set constants
IMG_SIZE = 224
IMG_HEIGHT = 224
IMG_WIDTH = 224
IMG_DEPTH = 3
In [ ]:
# split data into train, validation and test
X_train, X_test, y_train, y_test = train_test_split(images, masks, test_size=0.2)
X_val, X_test, y_val, y_test = train_test_split(X_test, y_test, test_size=0.2)

print(f"Shape of X_train is '{X_train.shape}' and the shape of y_train is '{y_train.shape}'")
print(f"Shape of X_val is '{X_val.shape}' and the shape of y_val is '{y_val.shape}'")
print(f"Shape of X_test is '{X_test.shape}' and the shape of y_test is '{y_test.shape}'")
Shape of X_train is '(327,)' and the shape of y_train is '(327,)'
Shape of X_val is '(65,)' and the shape of y_val is '(65,)'
Shape of X_test is '(17,)' and the shape of y_test is '(17,)'
In [ ]:
def get_images_masks(images,masks):

  # initialise arrays to hold data
  #Input image is size IMG_HEIGHT,IMG_WIDTH,IMG_DEPTH
  img_batch = np.zeros((images.shape[0], IMG_HEIGHT, IMG_WIDTH,IMG_DEPTH))
  #Mask's size is IMG_HEIGHT, IMG_WIDTH
  mask_batch = np.zeros((masks.shape[0], IMG_HEIGHT, IMG_WIDTH))

  #Populate X and y with actual data
  for i in range(images.shape[0]):

    #Resize image
    img = images[i]
    img = cv2.resize(img, dsize=(IMG_HEIGHT, IMG_WIDTH), interpolation=cv2.INTER_CUBIC)

    # assign all pixels in the first 3 channels only to the image, i.e., discard the alpha channel
    try:
      img = img[:,:,:3]
    except:
      # convert the grayscale image to color so that the number of channels are standardized to 3
      img = cv2.cvtColor(img, cv2.COLOR_GRAY2RGB)
      continue

    # pre-process input as per the pre-trained model used and add to the batch
    img_batch[i] = tf.keras.applications.mobilenet_v2.preprocess_input(np.array(img, dtype=np.float32))

    # Read Masks
    for dictionary in masks[i]:
      # get min mask co-ordinates
      xmin = int(dictionary['points'][0]['x'] * IMG_WIDTH)
      ymin = int(dictionary['points'][0]['y'] * IMG_HEIGHT)

      # get max mask co-ordinates
      xmax = int(dictionary['points'][1]['x'] * IMG_WIDTH)
      ymax = int(dictionary['points'][1]['y'] * IMG_HEIGHT)

      # set all pixels within the mask co-ordinates to 1.
      mask_batch[i][ymin:ymax,xmin:xmax] = 1
  return img_batch, mask_batch
In [ ]:
# Change to tensors
X_train , y_train = get_images_masks(X_train,y_train)
X_val, y_val = get_images_masks(X_val, y_val)
X_test, y_test = get_images_masks(X_test, y_test)

print(f"Shape of X_train is '{X_train.shape}' and the shape of y_train is '{y_train.shape}'")
print(f"Shape of X_val is '{X_val.shape}' and the shape of y_val is '{y_val.shape}'")
print(f"Shape of X_test is '{X_test.shape}' and the shape of y_test is '{y_test.shape}'")
Shape of X_train is '(327, 224, 224, 3)' and the shape of y_train is '(327, 224, 224)'
Shape of X_val is '(65, 224, 224, 3)' and the shape of y_val is '(65, 224, 224)'
Shape of X_test is '(17, 224, 224, 3)' and the shape of y_test is '(17, 224, 224)'
In [ ]:
# visualise masks and images for generator for verification

idx = np.random.randint(0,X_train.shape[0],2)
# initialising subplots
figure, ax = plt.subplots(nrows=2, ncols=2)

# setting figure parameters
figure.set_figheight(15)
figure.set_figwidth(15)

# setting images and masks to axis
ax[0][0].imshow(X_train[idx][0])
ax[1][0].imshow(y_train[idx][0])

ax[0][1].imshow(X_train[idx][1])
ax[1][1].imshow(y_train[idx][1])


plt.show()
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).

Observations:

  • Size of each image is 224x224x3 and each mask is 224x224.
  • We have 327 train images and corresponding masks.
  • We have 65 validation images and corresponding masks.
  • We have 17 testing images and corresponding masks.
  • The image lighting conditions are different and all correspond to rectangular masks for faces.

Model Building

We will be using Transfer Learning technique to get a pre-trained model(MobileNet in our case) and form a U-Net architecture for segmentation. Clearly we are going for semantic segmentation because all we are detecting is a face. The same problem can be extended for instance segmentation and a Mask R-CNN can be used.

The pre-trained MobileNet will be used as the encoder part of the U-Net Architecture. The decoder part will be built using upsampling layers.

In [ ]:
#Define input layer
input_tensor = tf.keras.layers.Input((IMG_HEIGHT, IMG_WIDTH, IMG_DEPTH), name='input_img')
In [ ]:
# download pre-trained model
pre_trained_model = tf.keras.applications.mobilenet_v2.MobileNetV2(input_tensor= input_tensor,
                                                                   alpha=0.5,
                                                                   weights='imagenet',
                                                                   include_top=False)
WARNING:tensorflow:`input_shape` is undefined or non-square, or `rows` is not in [96, 128, 160, 192, 224]. Weights for input shape (224, 224) will be loaded as the default.
Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/mobilenet_v2/mobilenet_v2_weights_tf_dim_ordering_tf_kernels_0.5_224_no_top.h5
3203072/3201480 [==============================] - 0s 0us/step
3211264/3201480 [==============================] - 0s 0us/step
In [ ]:
print(f'Number of layers in MobileNet are {len(pre_trained_model.layers)}')
Number of layers in MobileNet are 154
In [ ]:
# describing the pre trained model
pre_trained_model.summary()
Model: "mobilenetv2_0.50_224"
__________________________________________________________________________________________________
 Layer (type)                   Output Shape         Param #     Connected to                     
==================================================================================================
 input_img (InputLayer)         [(None, 224, 224, 3  0           []                               
                                )]                                                                
                                                                                                  
 Conv1 (Conv2D)                 (None, 112, 112, 16  432         ['input_img[0][0]']              
                                )                                                                 
                                                                                                  
 bn_Conv1 (BatchNormalization)  (None, 112, 112, 16  64          ['Conv1[0][0]']                  
                                )                                                                 
                                                                                                  
 Conv1_relu (ReLU)              (None, 112, 112, 16  0           ['bn_Conv1[0][0]']               
                                )                                                                 
                                                                                                  
 expanded_conv_depthwise (Depth  (None, 112, 112, 16  144        ['Conv1_relu[0][0]']             
 wiseConv2D)                    )                                                                 
                                                                                                  
 expanded_conv_depthwise_BN (Ba  (None, 112, 112, 16  64         ['expanded_conv_depthwise[0][0]']
 tchNormalization)              )                                                                 
                                                                                                  
 expanded_conv_depthwise_relu (  (None, 112, 112, 16  0          ['expanded_conv_depthwise_BN[0][0
 ReLU)                          )                                ]']                              
                                                                                                  
 expanded_conv_project (Conv2D)  (None, 112, 112, 8)  128        ['expanded_conv_depthwise_relu[0]
                                                                 [0]']                            
                                                                                                  
 expanded_conv_project_BN (Batc  (None, 112, 112, 8)  32         ['expanded_conv_project[0][0]']  
 hNormalization)                                                                                  
                                                                                                  
 block_1_expand (Conv2D)        (None, 112, 112, 48  384         ['expanded_conv_project_BN[0][0]'
                                )                                ]                                
                                                                                                  
 block_1_expand_BN (BatchNormal  (None, 112, 112, 48  192        ['block_1_expand[0][0]']         
 ization)                       )                                                                 
                                                                                                  
 block_1_expand_relu (ReLU)     (None, 112, 112, 48  0           ['block_1_expand_BN[0][0]']      
                                )                                                                 
                                                                                                  
 block_1_pad (ZeroPadding2D)    (None, 113, 113, 48  0           ['block_1_expand_relu[0][0]']    
                                )                                                                 
                                                                                                  
 block_1_depthwise (DepthwiseCo  (None, 56, 56, 48)  432         ['block_1_pad[0][0]']            
 nv2D)                                                                                            
                                                                                                  
 block_1_depthwise_BN (BatchNor  (None, 56, 56, 48)  192         ['block_1_depthwise[0][0]']      
 malization)                                                                                      
                                                                                                  
 block_1_depthwise_relu (ReLU)  (None, 56, 56, 48)   0           ['block_1_depthwise_BN[0][0]']   
                                                                                                  
 block_1_project (Conv2D)       (None, 56, 56, 16)   768         ['block_1_depthwise_relu[0][0]'] 
                                                                                                  
 block_1_project_BN (BatchNorma  (None, 56, 56, 16)  64          ['block_1_project[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_2_expand (Conv2D)        (None, 56, 56, 96)   1536        ['block_1_project_BN[0][0]']     
                                                                                                  
 block_2_expand_BN (BatchNormal  (None, 56, 56, 96)  384         ['block_2_expand[0][0]']         
 ization)                                                                                         
                                                                                                  
 block_2_expand_relu (ReLU)     (None, 56, 56, 96)   0           ['block_2_expand_BN[0][0]']      
                                                                                                  
 block_2_depthwise (DepthwiseCo  (None, 56, 56, 96)  864         ['block_2_expand_relu[0][0]']    
 nv2D)                                                                                            
                                                                                                  
 block_2_depthwise_BN (BatchNor  (None, 56, 56, 96)  384         ['block_2_depthwise[0][0]']      
 malization)                                                                                      
                                                                                                  
 block_2_depthwise_relu (ReLU)  (None, 56, 56, 96)   0           ['block_2_depthwise_BN[0][0]']   
                                                                                                  
 block_2_project (Conv2D)       (None, 56, 56, 16)   1536        ['block_2_depthwise_relu[0][0]'] 
                                                                                                  
 block_2_project_BN (BatchNorma  (None, 56, 56, 16)  64          ['block_2_project[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_2_add (Add)              (None, 56, 56, 16)   0           ['block_1_project_BN[0][0]',     
                                                                  'block_2_project_BN[0][0]']     
                                                                                                  
 block_3_expand (Conv2D)        (None, 56, 56, 96)   1536        ['block_2_add[0][0]']            
                                                                                                  
 block_3_expand_BN (BatchNormal  (None, 56, 56, 96)  384         ['block_3_expand[0][0]']         
 ization)                                                                                         
                                                                                                  
 block_3_expand_relu (ReLU)     (None, 56, 56, 96)   0           ['block_3_expand_BN[0][0]']      
                                                                                                  
 block_3_pad (ZeroPadding2D)    (None, 57, 57, 96)   0           ['block_3_expand_relu[0][0]']    
                                                                                                  
 block_3_depthwise (DepthwiseCo  (None, 28, 28, 96)  864         ['block_3_pad[0][0]']            
 nv2D)                                                                                            
                                                                                                  
 block_3_depthwise_BN (BatchNor  (None, 28, 28, 96)  384         ['block_3_depthwise[0][0]']      
 malization)                                                                                      
                                                                                                  
 block_3_depthwise_relu (ReLU)  (None, 28, 28, 96)   0           ['block_3_depthwise_BN[0][0]']   
                                                                                                  
 block_3_project (Conv2D)       (None, 28, 28, 16)   1536        ['block_3_depthwise_relu[0][0]'] 
                                                                                                  
 block_3_project_BN (BatchNorma  (None, 28, 28, 16)  64          ['block_3_project[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_4_expand (Conv2D)        (None, 28, 28, 96)   1536        ['block_3_project_BN[0][0]']     
                                                                                                  
 block_4_expand_BN (BatchNormal  (None, 28, 28, 96)  384         ['block_4_expand[0][0]']         
 ization)                                                                                         
                                                                                                  
 block_4_expand_relu (ReLU)     (None, 28, 28, 96)   0           ['block_4_expand_BN[0][0]']      
                                                                                                  
 block_4_depthwise (DepthwiseCo  (None, 28, 28, 96)  864         ['block_4_expand_relu[0][0]']    
 nv2D)                                                                                            
                                                                                                  
 block_4_depthwise_BN (BatchNor  (None, 28, 28, 96)  384         ['block_4_depthwise[0][0]']      
 malization)                                                                                      
                                                                                                  
 block_4_depthwise_relu (ReLU)  (None, 28, 28, 96)   0           ['block_4_depthwise_BN[0][0]']   
                                                                                                  
 block_4_project (Conv2D)       (None, 28, 28, 16)   1536        ['block_4_depthwise_relu[0][0]'] 
                                                                                                  
 block_4_project_BN (BatchNorma  (None, 28, 28, 16)  64          ['block_4_project[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_4_add (Add)              (None, 28, 28, 16)   0           ['block_3_project_BN[0][0]',     
                                                                  'block_4_project_BN[0][0]']     
                                                                                                  
 block_5_expand (Conv2D)        (None, 28, 28, 96)   1536        ['block_4_add[0][0]']            
                                                                                                  
 block_5_expand_BN (BatchNormal  (None, 28, 28, 96)  384         ['block_5_expand[0][0]']         
 ization)                                                                                         
                                                                                                  
 block_5_expand_relu (ReLU)     (None, 28, 28, 96)   0           ['block_5_expand_BN[0][0]']      
                                                                                                  
 block_5_depthwise (DepthwiseCo  (None, 28, 28, 96)  864         ['block_5_expand_relu[0][0]']    
 nv2D)                                                                                            
                                                                                                  
 block_5_depthwise_BN (BatchNor  (None, 28, 28, 96)  384         ['block_5_depthwise[0][0]']      
 malization)                                                                                      
                                                                                                  
 block_5_depthwise_relu (ReLU)  (None, 28, 28, 96)   0           ['block_5_depthwise_BN[0][0]']   
                                                                                                  
 block_5_project (Conv2D)       (None, 28, 28, 16)   1536        ['block_5_depthwise_relu[0][0]'] 
                                                                                                  
 block_5_project_BN (BatchNorma  (None, 28, 28, 16)  64          ['block_5_project[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_5_add (Add)              (None, 28, 28, 16)   0           ['block_4_add[0][0]',            
                                                                  'block_5_project_BN[0][0]']     
                                                                                                  
 block_6_expand (Conv2D)        (None, 28, 28, 96)   1536        ['block_5_add[0][0]']            
                                                                                                  
 block_6_expand_BN (BatchNormal  (None, 28, 28, 96)  384         ['block_6_expand[0][0]']         
 ization)                                                                                         
                                                                                                  
 block_6_expand_relu (ReLU)     (None, 28, 28, 96)   0           ['block_6_expand_BN[0][0]']      
                                                                                                  
 block_6_pad (ZeroPadding2D)    (None, 29, 29, 96)   0           ['block_6_expand_relu[0][0]']    
                                                                                                  
 block_6_depthwise (DepthwiseCo  (None, 14, 14, 96)  864         ['block_6_pad[0][0]']            
 nv2D)                                                                                            
                                                                                                  
 block_6_depthwise_BN (BatchNor  (None, 14, 14, 96)  384         ['block_6_depthwise[0][0]']      
 malization)                                                                                      
                                                                                                  
 block_6_depthwise_relu (ReLU)  (None, 14, 14, 96)   0           ['block_6_depthwise_BN[0][0]']   
                                                                                                  
 block_6_project (Conv2D)       (None, 14, 14, 32)   3072        ['block_6_depthwise_relu[0][0]'] 
                                                                                                  
 block_6_project_BN (BatchNorma  (None, 14, 14, 32)  128         ['block_6_project[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_7_expand (Conv2D)        (None, 14, 14, 192)  6144        ['block_6_project_BN[0][0]']     
                                                                                                  
 block_7_expand_BN (BatchNormal  (None, 14, 14, 192)  768        ['block_7_expand[0][0]']         
 ization)                                                                                         
                                                                                                  
 block_7_expand_relu (ReLU)     (None, 14, 14, 192)  0           ['block_7_expand_BN[0][0]']      
                                                                                                  
 block_7_depthwise (DepthwiseCo  (None, 14, 14, 192)  1728       ['block_7_expand_relu[0][0]']    
 nv2D)                                                                                            
                                                                                                  
 block_7_depthwise_BN (BatchNor  (None, 14, 14, 192)  768        ['block_7_depthwise[0][0]']      
 malization)                                                                                      
                                                                                                  
 block_7_depthwise_relu (ReLU)  (None, 14, 14, 192)  0           ['block_7_depthwise_BN[0][0]']   
                                                                                                  
 block_7_project (Conv2D)       (None, 14, 14, 32)   6144        ['block_7_depthwise_relu[0][0]'] 
                                                                                                  
 block_7_project_BN (BatchNorma  (None, 14, 14, 32)  128         ['block_7_project[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_7_add (Add)              (None, 14, 14, 32)   0           ['block_6_project_BN[0][0]',     
                                                                  'block_7_project_BN[0][0]']     
                                                                                                  
 block_8_expand (Conv2D)        (None, 14, 14, 192)  6144        ['block_7_add[0][0]']            
                                                                                                  
 block_8_expand_BN (BatchNormal  (None, 14, 14, 192)  768        ['block_8_expand[0][0]']         
 ization)                                                                                         
                                                                                                  
 block_8_expand_relu (ReLU)     (None, 14, 14, 192)  0           ['block_8_expand_BN[0][0]']      
                                                                                                  
 block_8_depthwise (DepthwiseCo  (None, 14, 14, 192)  1728       ['block_8_expand_relu[0][0]']    
 nv2D)                                                                                            
                                                                                                  
 block_8_depthwise_BN (BatchNor  (None, 14, 14, 192)  768        ['block_8_depthwise[0][0]']      
 malization)                                                                                      
                                                                                                  
 block_8_depthwise_relu (ReLU)  (None, 14, 14, 192)  0           ['block_8_depthwise_BN[0][0]']   
                                                                                                  
 block_8_project (Conv2D)       (None, 14, 14, 32)   6144        ['block_8_depthwise_relu[0][0]'] 
                                                                                                  
 block_8_project_BN (BatchNorma  (None, 14, 14, 32)  128         ['block_8_project[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_8_add (Add)              (None, 14, 14, 32)   0           ['block_7_add[0][0]',            
                                                                  'block_8_project_BN[0][0]']     
                                                                                                  
 block_9_expand (Conv2D)        (None, 14, 14, 192)  6144        ['block_8_add[0][0]']            
                                                                                                  
 block_9_expand_BN (BatchNormal  (None, 14, 14, 192)  768        ['block_9_expand[0][0]']         
 ization)                                                                                         
                                                                                                  
 block_9_expand_relu (ReLU)     (None, 14, 14, 192)  0           ['block_9_expand_BN[0][0]']      
                                                                                                  
 block_9_depthwise (DepthwiseCo  (None, 14, 14, 192)  1728       ['block_9_expand_relu[0][0]']    
 nv2D)                                                                                            
                                                                                                  
 block_9_depthwise_BN (BatchNor  (None, 14, 14, 192)  768        ['block_9_depthwise[0][0]']      
 malization)                                                                                      
                                                                                                  
 block_9_depthwise_relu (ReLU)  (None, 14, 14, 192)  0           ['block_9_depthwise_BN[0][0]']   
                                                                                                  
 block_9_project (Conv2D)       (None, 14, 14, 32)   6144        ['block_9_depthwise_relu[0][0]'] 
                                                                                                  
 block_9_project_BN (BatchNorma  (None, 14, 14, 32)  128         ['block_9_project[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_9_add (Add)              (None, 14, 14, 32)   0           ['block_8_add[0][0]',            
                                                                  'block_9_project_BN[0][0]']     
                                                                                                  
 block_10_expand (Conv2D)       (None, 14, 14, 192)  6144        ['block_9_add[0][0]']            
                                                                                                  
 block_10_expand_BN (BatchNorma  (None, 14, 14, 192)  768        ['block_10_expand[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_10_expand_relu (ReLU)    (None, 14, 14, 192)  0           ['block_10_expand_BN[0][0]']     
                                                                                                  
 block_10_depthwise (DepthwiseC  (None, 14, 14, 192)  1728       ['block_10_expand_relu[0][0]']   
 onv2D)                                                                                           
                                                                                                  
 block_10_depthwise_BN (BatchNo  (None, 14, 14, 192)  768        ['block_10_depthwise[0][0]']     
 rmalization)                                                                                     
                                                                                                  
 block_10_depthwise_relu (ReLU)  (None, 14, 14, 192)  0          ['block_10_depthwise_BN[0][0]']  
                                                                                                  
 block_10_project (Conv2D)      (None, 14, 14, 48)   9216        ['block_10_depthwise_relu[0][0]']
                                                                                                  
 block_10_project_BN (BatchNorm  (None, 14, 14, 48)  192         ['block_10_project[0][0]']       
 alization)                                                                                       
                                                                                                  
 block_11_expand (Conv2D)       (None, 14, 14, 288)  13824       ['block_10_project_BN[0][0]']    
                                                                                                  
 block_11_expand_BN (BatchNorma  (None, 14, 14, 288)  1152       ['block_11_expand[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_11_expand_relu (ReLU)    (None, 14, 14, 288)  0           ['block_11_expand_BN[0][0]']     
                                                                                                  
 block_11_depthwise (DepthwiseC  (None, 14, 14, 288)  2592       ['block_11_expand_relu[0][0]']   
 onv2D)                                                                                           
                                                                                                  
 block_11_depthwise_BN (BatchNo  (None, 14, 14, 288)  1152       ['block_11_depthwise[0][0]']     
 rmalization)                                                                                     
                                                                                                  
 block_11_depthwise_relu (ReLU)  (None, 14, 14, 288)  0          ['block_11_depthwise_BN[0][0]']  
                                                                                                  
 block_11_project (Conv2D)      (None, 14, 14, 48)   13824       ['block_11_depthwise_relu[0][0]']
                                                                                                  
 block_11_project_BN (BatchNorm  (None, 14, 14, 48)  192         ['block_11_project[0][0]']       
 alization)                                                                                       
                                                                                                  
 block_11_add (Add)             (None, 14, 14, 48)   0           ['block_10_project_BN[0][0]',    
                                                                  'block_11_project_BN[0][0]']    
                                                                                                  
 block_12_expand (Conv2D)       (None, 14, 14, 288)  13824       ['block_11_add[0][0]']           
                                                                                                  
 block_12_expand_BN (BatchNorma  (None, 14, 14, 288)  1152       ['block_12_expand[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_12_expand_relu (ReLU)    (None, 14, 14, 288)  0           ['block_12_expand_BN[0][0]']     
                                                                                                  
 block_12_depthwise (DepthwiseC  (None, 14, 14, 288)  2592       ['block_12_expand_relu[0][0]']   
 onv2D)                                                                                           
                                                                                                  
 block_12_depthwise_BN (BatchNo  (None, 14, 14, 288)  1152       ['block_12_depthwise[0][0]']     
 rmalization)                                                                                     
                                                                                                  
 block_12_depthwise_relu (ReLU)  (None, 14, 14, 288)  0          ['block_12_depthwise_BN[0][0]']  
                                                                                                  
 block_12_project (Conv2D)      (None, 14, 14, 48)   13824       ['block_12_depthwise_relu[0][0]']
                                                                                                  
 block_12_project_BN (BatchNorm  (None, 14, 14, 48)  192         ['block_12_project[0][0]']       
 alization)                                                                                       
                                                                                                  
 block_12_add (Add)             (None, 14, 14, 48)   0           ['block_11_add[0][0]',           
                                                                  'block_12_project_BN[0][0]']    
                                                                                                  
 block_13_expand (Conv2D)       (None, 14, 14, 288)  13824       ['block_12_add[0][0]']           
                                                                                                  
 block_13_expand_BN (BatchNorma  (None, 14, 14, 288)  1152       ['block_13_expand[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_13_expand_relu (ReLU)    (None, 14, 14, 288)  0           ['block_13_expand_BN[0][0]']     
                                                                                                  
 block_13_pad (ZeroPadding2D)   (None, 15, 15, 288)  0           ['block_13_expand_relu[0][0]']   
                                                                                                  
 block_13_depthwise (DepthwiseC  (None, 7, 7, 288)   2592        ['block_13_pad[0][0]']           
 onv2D)                                                                                           
                                                                                                  
 block_13_depthwise_BN (BatchNo  (None, 7, 7, 288)   1152        ['block_13_depthwise[0][0]']     
 rmalization)                                                                                     
                                                                                                  
 block_13_depthwise_relu (ReLU)  (None, 7, 7, 288)   0           ['block_13_depthwise_BN[0][0]']  
                                                                                                  
 block_13_project (Conv2D)      (None, 7, 7, 80)     23040       ['block_13_depthwise_relu[0][0]']
                                                                                                  
 block_13_project_BN (BatchNorm  (None, 7, 7, 80)    320         ['block_13_project[0][0]']       
 alization)                                                                                       
                                                                                                  
 block_14_expand (Conv2D)       (None, 7, 7, 480)    38400       ['block_13_project_BN[0][0]']    
                                                                                                  
 block_14_expand_BN (BatchNorma  (None, 7, 7, 480)   1920        ['block_14_expand[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_14_expand_relu (ReLU)    (None, 7, 7, 480)    0           ['block_14_expand_BN[0][0]']     
                                                                                                  
 block_14_depthwise (DepthwiseC  (None, 7, 7, 480)   4320        ['block_14_expand_relu[0][0]']   
 onv2D)                                                                                           
                                                                                                  
 block_14_depthwise_BN (BatchNo  (None, 7, 7, 480)   1920        ['block_14_depthwise[0][0]']     
 rmalization)                                                                                     
                                                                                                  
 block_14_depthwise_relu (ReLU)  (None, 7, 7, 480)   0           ['block_14_depthwise_BN[0][0]']  
                                                                                                  
 block_14_project (Conv2D)      (None, 7, 7, 80)     38400       ['block_14_depthwise_relu[0][0]']
                                                                                                  
 block_14_project_BN (BatchNorm  (None, 7, 7, 80)    320         ['block_14_project[0][0]']       
 alization)                                                                                       
                                                                                                  
 block_14_add (Add)             (None, 7, 7, 80)     0           ['block_13_project_BN[0][0]',    
                                                                  'block_14_project_BN[0][0]']    
                                                                                                  
 block_15_expand (Conv2D)       (None, 7, 7, 480)    38400       ['block_14_add[0][0]']           
                                                                                                  
 block_15_expand_BN (BatchNorma  (None, 7, 7, 480)   1920        ['block_15_expand[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_15_expand_relu (ReLU)    (None, 7, 7, 480)    0           ['block_15_expand_BN[0][0]']     
                                                                                                  
 block_15_depthwise (DepthwiseC  (None, 7, 7, 480)   4320        ['block_15_expand_relu[0][0]']   
 onv2D)                                                                                           
                                                                                                  
 block_15_depthwise_BN (BatchNo  (None, 7, 7, 480)   1920        ['block_15_depthwise[0][0]']     
 rmalization)                                                                                     
                                                                                                  
 block_15_depthwise_relu (ReLU)  (None, 7, 7, 480)   0           ['block_15_depthwise_BN[0][0]']  
                                                                                                  
 block_15_project (Conv2D)      (None, 7, 7, 80)     38400       ['block_15_depthwise_relu[0][0]']
                                                                                                  
 block_15_project_BN (BatchNorm  (None, 7, 7, 80)    320         ['block_15_project[0][0]']       
 alization)                                                                                       
                                                                                                  
 block_15_add (Add)             (None, 7, 7, 80)     0           ['block_14_add[0][0]',           
                                                                  'block_15_project_BN[0][0]']    
                                                                                                  
 block_16_expand (Conv2D)       (None, 7, 7, 480)    38400       ['block_15_add[0][0]']           
                                                                                                  
 block_16_expand_BN (BatchNorma  (None, 7, 7, 480)   1920        ['block_16_expand[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_16_expand_relu (ReLU)    (None, 7, 7, 480)    0           ['block_16_expand_BN[0][0]']     
                                                                                                  
 block_16_depthwise (DepthwiseC  (None, 7, 7, 480)   4320        ['block_16_expand_relu[0][0]']   
 onv2D)                                                                                           
                                                                                                  
 block_16_depthwise_BN (BatchNo  (None, 7, 7, 480)   1920        ['block_16_depthwise[0][0]']     
 rmalization)                                                                                     
                                                                                                  
 block_16_depthwise_relu (ReLU)  (None, 7, 7, 480)   0           ['block_16_depthwise_BN[0][0]']  
                                                                                                  
 block_16_project (Conv2D)      (None, 7, 7, 160)    76800       ['block_16_depthwise_relu[0][0]']
                                                                                                  
 block_16_project_BN (BatchNorm  (None, 7, 7, 160)   640         ['block_16_project[0][0]']       
 alization)                                                                                       
                                                                                                  
 Conv_1 (Conv2D)                (None, 7, 7, 1280)   204800      ['block_16_project_BN[0][0]']    
                                                                                                  
 Conv_1_bn (BatchNormalization)  (None, 7, 7, 1280)  5120        ['Conv_1[0][0]']                 
                                                                                                  
 out_relu (ReLU)                (None, 7, 7, 1280)   0           ['Conv_1_bn[0][0]']              
                                                                                                  
==================================================================================================
Total params: 706,224
Trainable params: 687,680
Non-trainable params: 18,544
__________________________________________________________________________________________________
In [ ]:
def conv2d_block(input_tensor, n_filters):
    """Function to add 2 convolutional layers with the parameters passed to it"""
    # first layer
    x = tf.keras.layers.Conv2D(n_filters, (3,3), kernel_initializer = 'he_normal', padding = 'same')(input_tensor)
    x = tf.keras.layers.BatchNormalization()(x)
    x = tf.keras.layers.Activation('relu')(x)
    
    # second layer
    x = tf.keras.layers.Conv2D(n_filters, (3,3), kernel_initializer = 'he_normal', padding = 'same')(x)
    x = tf.keras.layers.BatchNormalization()(x)
    x = tf.keras.layers.Activation('relu')(x)
    
    return x
In [ ]:
def build_decoder_block(input, layer_name, number_of_filters):
  layer_output = pre_trained_model.get_layer(layer_name).output
  x = tf.keras.layers.UpSampling2D()(input)
  #Skip connection
  x = tf.keras.layers.Concatenate()([x, layer_output])
  x = conv2d_block(x,number_of_filters)
  return x
In [ ]:
def build_decoder():
  
  skip_connection_names = ["input_img", "block_1_expand_relu", "block_3_expand_relu", "block_6_expand_relu"]
  encoder_output = pre_trained_model.get_layer('block_13_expand_relu').output
  filters = [16, 32, 48, 64]

  x = encoder_output
  for i in range(1, len(skip_connection_names)+1):

    # Decoder blocks
    layer_name = skip_connection_names[-i]
    number_of_filters = filters[-i]
    x = build_decoder_block(x,layer_name,number_of_filters)

  return x
In [ ]:
decoder = build_decoder()
In [ ]:
def build_model(input,decoder):
  #Build the Output layer
  outputs = tf.keras.layers.Conv2D(1, (1, 1), activation='sigmoid')(decoder)

  #Build the model using different layers
  model = tf.keras.Model(inputs=[input], outputs=[outputs])
  return model
In [ ]:
final_model = build_model(pre_trained_model.input,decoder)
In [ ]:
final_model.summary()
Model: "model"
__________________________________________________________________________________________________
 Layer (type)                   Output Shape         Param #     Connected to                     
==================================================================================================
 input_img (InputLayer)         [(None, 224, 224, 3  0           []                               
                                )]                                                                
                                                                                                  
 Conv1 (Conv2D)                 (None, 112, 112, 16  432         ['input_img[0][0]']              
                                )                                                                 
                                                                                                  
 bn_Conv1 (BatchNormalization)  (None, 112, 112, 16  64          ['Conv1[0][0]']                  
                                )                                                                 
                                                                                                  
 Conv1_relu (ReLU)              (None, 112, 112, 16  0           ['bn_Conv1[0][0]']               
                                )                                                                 
                                                                                                  
 expanded_conv_depthwise (Depth  (None, 112, 112, 16  144        ['Conv1_relu[0][0]']             
 wiseConv2D)                    )                                                                 
                                                                                                  
 expanded_conv_depthwise_BN (Ba  (None, 112, 112, 16  64         ['expanded_conv_depthwise[0][0]']
 tchNormalization)              )                                                                 
                                                                                                  
 expanded_conv_depthwise_relu (  (None, 112, 112, 16  0          ['expanded_conv_depthwise_BN[0][0
 ReLU)                          )                                ]']                              
                                                                                                  
 expanded_conv_project (Conv2D)  (None, 112, 112, 8)  128        ['expanded_conv_depthwise_relu[0]
                                                                 [0]']                            
                                                                                                  
 expanded_conv_project_BN (Batc  (None, 112, 112, 8)  32         ['expanded_conv_project[0][0]']  
 hNormalization)                                                                                  
                                                                                                  
 block_1_expand (Conv2D)        (None, 112, 112, 48  384         ['expanded_conv_project_BN[0][0]'
                                )                                ]                                
                                                                                                  
 block_1_expand_BN (BatchNormal  (None, 112, 112, 48  192        ['block_1_expand[0][0]']         
 ization)                       )                                                                 
                                                                                                  
 block_1_expand_relu (ReLU)     (None, 112, 112, 48  0           ['block_1_expand_BN[0][0]']      
                                )                                                                 
                                                                                                  
 block_1_pad (ZeroPadding2D)    (None, 113, 113, 48  0           ['block_1_expand_relu[0][0]']    
                                )                                                                 
                                                                                                  
 block_1_depthwise (DepthwiseCo  (None, 56, 56, 48)  432         ['block_1_pad[0][0]']            
 nv2D)                                                                                            
                                                                                                  
 block_1_depthwise_BN (BatchNor  (None, 56, 56, 48)  192         ['block_1_depthwise[0][0]']      
 malization)                                                                                      
                                                                                                  
 block_1_depthwise_relu (ReLU)  (None, 56, 56, 48)   0           ['block_1_depthwise_BN[0][0]']   
                                                                                                  
 block_1_project (Conv2D)       (None, 56, 56, 16)   768         ['block_1_depthwise_relu[0][0]'] 
                                                                                                  
 block_1_project_BN (BatchNorma  (None, 56, 56, 16)  64          ['block_1_project[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_2_expand (Conv2D)        (None, 56, 56, 96)   1536        ['block_1_project_BN[0][0]']     
                                                                                                  
 block_2_expand_BN (BatchNormal  (None, 56, 56, 96)  384         ['block_2_expand[0][0]']         
 ization)                                                                                         
                                                                                                  
 block_2_expand_relu (ReLU)     (None, 56, 56, 96)   0           ['block_2_expand_BN[0][0]']      
                                                                                                  
 block_2_depthwise (DepthwiseCo  (None, 56, 56, 96)  864         ['block_2_expand_relu[0][0]']    
 nv2D)                                                                                            
                                                                                                  
 block_2_depthwise_BN (BatchNor  (None, 56, 56, 96)  384         ['block_2_depthwise[0][0]']      
 malization)                                                                                      
                                                                                                  
 block_2_depthwise_relu (ReLU)  (None, 56, 56, 96)   0           ['block_2_depthwise_BN[0][0]']   
                                                                                                  
 block_2_project (Conv2D)       (None, 56, 56, 16)   1536        ['block_2_depthwise_relu[0][0]'] 
                                                                                                  
 block_2_project_BN (BatchNorma  (None, 56, 56, 16)  64          ['block_2_project[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_2_add (Add)              (None, 56, 56, 16)   0           ['block_1_project_BN[0][0]',     
                                                                  'block_2_project_BN[0][0]']     
                                                                                                  
 block_3_expand (Conv2D)        (None, 56, 56, 96)   1536        ['block_2_add[0][0]']            
                                                                                                  
 block_3_expand_BN (BatchNormal  (None, 56, 56, 96)  384         ['block_3_expand[0][0]']         
 ization)                                                                                         
                                                                                                  
 block_3_expand_relu (ReLU)     (None, 56, 56, 96)   0           ['block_3_expand_BN[0][0]']      
                                                                                                  
 block_3_pad (ZeroPadding2D)    (None, 57, 57, 96)   0           ['block_3_expand_relu[0][0]']    
                                                                                                  
 block_3_depthwise (DepthwiseCo  (None, 28, 28, 96)  864         ['block_3_pad[0][0]']            
 nv2D)                                                                                            
                                                                                                  
 block_3_depthwise_BN (BatchNor  (None, 28, 28, 96)  384         ['block_3_depthwise[0][0]']      
 malization)                                                                                      
                                                                                                  
 block_3_depthwise_relu (ReLU)  (None, 28, 28, 96)   0           ['block_3_depthwise_BN[0][0]']   
                                                                                                  
 block_3_project (Conv2D)       (None, 28, 28, 16)   1536        ['block_3_depthwise_relu[0][0]'] 
                                                                                                  
 block_3_project_BN (BatchNorma  (None, 28, 28, 16)  64          ['block_3_project[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_4_expand (Conv2D)        (None, 28, 28, 96)   1536        ['block_3_project_BN[0][0]']     
                                                                                                  
 block_4_expand_BN (BatchNormal  (None, 28, 28, 96)  384         ['block_4_expand[0][0]']         
 ization)                                                                                         
                                                                                                  
 block_4_expand_relu (ReLU)     (None, 28, 28, 96)   0           ['block_4_expand_BN[0][0]']      
                                                                                                  
 block_4_depthwise (DepthwiseCo  (None, 28, 28, 96)  864         ['block_4_expand_relu[0][0]']    
 nv2D)                                                                                            
                                                                                                  
 block_4_depthwise_BN (BatchNor  (None, 28, 28, 96)  384         ['block_4_depthwise[0][0]']      
 malization)                                                                                      
                                                                                                  
 block_4_depthwise_relu (ReLU)  (None, 28, 28, 96)   0           ['block_4_depthwise_BN[0][0]']   
                                                                                                  
 block_4_project (Conv2D)       (None, 28, 28, 16)   1536        ['block_4_depthwise_relu[0][0]'] 
                                                                                                  
 block_4_project_BN (BatchNorma  (None, 28, 28, 16)  64          ['block_4_project[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_4_add (Add)              (None, 28, 28, 16)   0           ['block_3_project_BN[0][0]',     
                                                                  'block_4_project_BN[0][0]']     
                                                                                                  
 block_5_expand (Conv2D)        (None, 28, 28, 96)   1536        ['block_4_add[0][0]']            
                                                                                                  
 block_5_expand_BN (BatchNormal  (None, 28, 28, 96)  384         ['block_5_expand[0][0]']         
 ization)                                                                                         
                                                                                                  
 block_5_expand_relu (ReLU)     (None, 28, 28, 96)   0           ['block_5_expand_BN[0][0]']      
                                                                                                  
 block_5_depthwise (DepthwiseCo  (None, 28, 28, 96)  864         ['block_5_expand_relu[0][0]']    
 nv2D)                                                                                            
                                                                                                  
 block_5_depthwise_BN (BatchNor  (None, 28, 28, 96)  384         ['block_5_depthwise[0][0]']      
 malization)                                                                                      
                                                                                                  
 block_5_depthwise_relu (ReLU)  (None, 28, 28, 96)   0           ['block_5_depthwise_BN[0][0]']   
                                                                                                  
 block_5_project (Conv2D)       (None, 28, 28, 16)   1536        ['block_5_depthwise_relu[0][0]'] 
                                                                                                  
 block_5_project_BN (BatchNorma  (None, 28, 28, 16)  64          ['block_5_project[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_5_add (Add)              (None, 28, 28, 16)   0           ['block_4_add[0][0]',            
                                                                  'block_5_project_BN[0][0]']     
                                                                                                  
 block_6_expand (Conv2D)        (None, 28, 28, 96)   1536        ['block_5_add[0][0]']            
                                                                                                  
 block_6_expand_BN (BatchNormal  (None, 28, 28, 96)  384         ['block_6_expand[0][0]']         
 ization)                                                                                         
                                                                                                  
 block_6_expand_relu (ReLU)     (None, 28, 28, 96)   0           ['block_6_expand_BN[0][0]']      
                                                                                                  
 block_6_pad (ZeroPadding2D)    (None, 29, 29, 96)   0           ['block_6_expand_relu[0][0]']    
                                                                                                  
 block_6_depthwise (DepthwiseCo  (None, 14, 14, 96)  864         ['block_6_pad[0][0]']            
 nv2D)                                                                                            
                                                                                                  
 block_6_depthwise_BN (BatchNor  (None, 14, 14, 96)  384         ['block_6_depthwise[0][0]']      
 malization)                                                                                      
                                                                                                  
 block_6_depthwise_relu (ReLU)  (None, 14, 14, 96)   0           ['block_6_depthwise_BN[0][0]']   
                                                                                                  
 block_6_project (Conv2D)       (None, 14, 14, 32)   3072        ['block_6_depthwise_relu[0][0]'] 
                                                                                                  
 block_6_project_BN (BatchNorma  (None, 14, 14, 32)  128         ['block_6_project[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_7_expand (Conv2D)        (None, 14, 14, 192)  6144        ['block_6_project_BN[0][0]']     
                                                                                                  
 block_7_expand_BN (BatchNormal  (None, 14, 14, 192)  768        ['block_7_expand[0][0]']         
 ization)                                                                                         
                                                                                                  
 block_7_expand_relu (ReLU)     (None, 14, 14, 192)  0           ['block_7_expand_BN[0][0]']      
                                                                                                  
 block_7_depthwise (DepthwiseCo  (None, 14, 14, 192)  1728       ['block_7_expand_relu[0][0]']    
 nv2D)                                                                                            
                                                                                                  
 block_7_depthwise_BN (BatchNor  (None, 14, 14, 192)  768        ['block_7_depthwise[0][0]']      
 malization)                                                                                      
                                                                                                  
 block_7_depthwise_relu (ReLU)  (None, 14, 14, 192)  0           ['block_7_depthwise_BN[0][0]']   
                                                                                                  
 block_7_project (Conv2D)       (None, 14, 14, 32)   6144        ['block_7_depthwise_relu[0][0]'] 
                                                                                                  
 block_7_project_BN (BatchNorma  (None, 14, 14, 32)  128         ['block_7_project[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_7_add (Add)              (None, 14, 14, 32)   0           ['block_6_project_BN[0][0]',     
                                                                  'block_7_project_BN[0][0]']     
                                                                                                  
 block_8_expand (Conv2D)        (None, 14, 14, 192)  6144        ['block_7_add[0][0]']            
                                                                                                  
 block_8_expand_BN (BatchNormal  (None, 14, 14, 192)  768        ['block_8_expand[0][0]']         
 ization)                                                                                         
                                                                                                  
 block_8_expand_relu (ReLU)     (None, 14, 14, 192)  0           ['block_8_expand_BN[0][0]']      
                                                                                                  
 block_8_depthwise (DepthwiseCo  (None, 14, 14, 192)  1728       ['block_8_expand_relu[0][0]']    
 nv2D)                                                                                            
                                                                                                  
 block_8_depthwise_BN (BatchNor  (None, 14, 14, 192)  768        ['block_8_depthwise[0][0]']      
 malization)                                                                                      
                                                                                                  
 block_8_depthwise_relu (ReLU)  (None, 14, 14, 192)  0           ['block_8_depthwise_BN[0][0]']   
                                                                                                  
 block_8_project (Conv2D)       (None, 14, 14, 32)   6144        ['block_8_depthwise_relu[0][0]'] 
                                                                                                  
 block_8_project_BN (BatchNorma  (None, 14, 14, 32)  128         ['block_8_project[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_8_add (Add)              (None, 14, 14, 32)   0           ['block_7_add[0][0]',            
                                                                  'block_8_project_BN[0][0]']     
                                                                                                  
 block_9_expand (Conv2D)        (None, 14, 14, 192)  6144        ['block_8_add[0][0]']            
                                                                                                  
 block_9_expand_BN (BatchNormal  (None, 14, 14, 192)  768        ['block_9_expand[0][0]']         
 ization)                                                                                         
                                                                                                  
 block_9_expand_relu (ReLU)     (None, 14, 14, 192)  0           ['block_9_expand_BN[0][0]']      
                                                                                                  
 block_9_depthwise (DepthwiseCo  (None, 14, 14, 192)  1728       ['block_9_expand_relu[0][0]']    
 nv2D)                                                                                            
                                                                                                  
 block_9_depthwise_BN (BatchNor  (None, 14, 14, 192)  768        ['block_9_depthwise[0][0]']      
 malization)                                                                                      
                                                                                                  
 block_9_depthwise_relu (ReLU)  (None, 14, 14, 192)  0           ['block_9_depthwise_BN[0][0]']   
                                                                                                  
 block_9_project (Conv2D)       (None, 14, 14, 32)   6144        ['block_9_depthwise_relu[0][0]'] 
                                                                                                  
 block_9_project_BN (BatchNorma  (None, 14, 14, 32)  128         ['block_9_project[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_9_add (Add)              (None, 14, 14, 32)   0           ['block_8_add[0][0]',            
                                                                  'block_9_project_BN[0][0]']     
                                                                                                  
 block_10_expand (Conv2D)       (None, 14, 14, 192)  6144        ['block_9_add[0][0]']            
                                                                                                  
 block_10_expand_BN (BatchNorma  (None, 14, 14, 192)  768        ['block_10_expand[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_10_expand_relu (ReLU)    (None, 14, 14, 192)  0           ['block_10_expand_BN[0][0]']     
                                                                                                  
 block_10_depthwise (DepthwiseC  (None, 14, 14, 192)  1728       ['block_10_expand_relu[0][0]']   
 onv2D)                                                                                           
                                                                                                  
 block_10_depthwise_BN (BatchNo  (None, 14, 14, 192)  768        ['block_10_depthwise[0][0]']     
 rmalization)                                                                                     
                                                                                                  
 block_10_depthwise_relu (ReLU)  (None, 14, 14, 192)  0          ['block_10_depthwise_BN[0][0]']  
                                                                                                  
 block_10_project (Conv2D)      (None, 14, 14, 48)   9216        ['block_10_depthwise_relu[0][0]']
                                                                                                  
 block_10_project_BN (BatchNorm  (None, 14, 14, 48)  192         ['block_10_project[0][0]']       
 alization)                                                                                       
                                                                                                  
 block_11_expand (Conv2D)       (None, 14, 14, 288)  13824       ['block_10_project_BN[0][0]']    
                                                                                                  
 block_11_expand_BN (BatchNorma  (None, 14, 14, 288)  1152       ['block_11_expand[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_11_expand_relu (ReLU)    (None, 14, 14, 288)  0           ['block_11_expand_BN[0][0]']     
                                                                                                  
 block_11_depthwise (DepthwiseC  (None, 14, 14, 288)  2592       ['block_11_expand_relu[0][0]']   
 onv2D)                                                                                           
                                                                                                  
 block_11_depthwise_BN (BatchNo  (None, 14, 14, 288)  1152       ['block_11_depthwise[0][0]']     
 rmalization)                                                                                     
                                                                                                  
 block_11_depthwise_relu (ReLU)  (None, 14, 14, 288)  0          ['block_11_depthwise_BN[0][0]']  
                                                                                                  
 block_11_project (Conv2D)      (None, 14, 14, 48)   13824       ['block_11_depthwise_relu[0][0]']
                                                                                                  
 block_11_project_BN (BatchNorm  (None, 14, 14, 48)  192         ['block_11_project[0][0]']       
 alization)                                                                                       
                                                                                                  
 block_11_add (Add)             (None, 14, 14, 48)   0           ['block_10_project_BN[0][0]',    
                                                                  'block_11_project_BN[0][0]']    
                                                                                                  
 block_12_expand (Conv2D)       (None, 14, 14, 288)  13824       ['block_11_add[0][0]']           
                                                                                                  
 block_12_expand_BN (BatchNorma  (None, 14, 14, 288)  1152       ['block_12_expand[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_12_expand_relu (ReLU)    (None, 14, 14, 288)  0           ['block_12_expand_BN[0][0]']     
                                                                                                  
 block_12_depthwise (DepthwiseC  (None, 14, 14, 288)  2592       ['block_12_expand_relu[0][0]']   
 onv2D)                                                                                           
                                                                                                  
 block_12_depthwise_BN (BatchNo  (None, 14, 14, 288)  1152       ['block_12_depthwise[0][0]']     
 rmalization)                                                                                     
                                                                                                  
 block_12_depthwise_relu (ReLU)  (None, 14, 14, 288)  0          ['block_12_depthwise_BN[0][0]']  
                                                                                                  
 block_12_project (Conv2D)      (None, 14, 14, 48)   13824       ['block_12_depthwise_relu[0][0]']
                                                                                                  
 block_12_project_BN (BatchNorm  (None, 14, 14, 48)  192         ['block_12_project[0][0]']       
 alization)                                                                                       
                                                                                                  
 block_12_add (Add)             (None, 14, 14, 48)   0           ['block_11_add[0][0]',           
                                                                  'block_12_project_BN[0][0]']    
                                                                                                  
 block_13_expand (Conv2D)       (None, 14, 14, 288)  13824       ['block_12_add[0][0]']           
                                                                                                  
 block_13_expand_BN (BatchNorma  (None, 14, 14, 288)  1152       ['block_13_expand[0][0]']        
 lization)                                                                                        
                                                                                                  
 block_13_expand_relu (ReLU)    (None, 14, 14, 288)  0           ['block_13_expand_BN[0][0]']     
                                                                                                  
 up_sampling2d (UpSampling2D)   (None, 28, 28, 288)  0           ['block_13_expand_relu[0][0]']   
                                                                                                  
 concatenate (Concatenate)      (None, 28, 28, 384)  0           ['up_sampling2d[0][0]',          
                                                                  'block_6_expand_relu[0][0]']    
                                                                                                  
 conv2d (Conv2D)                (None, 28, 28, 64)   221248      ['concatenate[0][0]']            
                                                                                                  
 batch_normalization (BatchNorm  (None, 28, 28, 64)  256         ['conv2d[0][0]']                 
 alization)                                                                                       
                                                                                                  
 activation (Activation)        (None, 28, 28, 64)   0           ['batch_normalization[0][0]']    
                                                                                                  
 conv2d_1 (Conv2D)              (None, 28, 28, 64)   36928       ['activation[0][0]']             
                                                                                                  
 batch_normalization_1 (BatchNo  (None, 28, 28, 64)  256         ['conv2d_1[0][0]']               
 rmalization)                                                                                     
                                                                                                  
 activation_1 (Activation)      (None, 28, 28, 64)   0           ['batch_normalization_1[0][0]']  
                                                                                                  
 up_sampling2d_1 (UpSampling2D)  (None, 56, 56, 64)  0           ['activation_1[0][0]']           
                                                                                                  
 concatenate_1 (Concatenate)    (None, 56, 56, 160)  0           ['up_sampling2d_1[0][0]',        
                                                                  'block_3_expand_relu[0][0]']    
                                                                                                  
 conv2d_2 (Conv2D)              (None, 56, 56, 48)   69168       ['concatenate_1[0][0]']          
                                                                                                  
 batch_normalization_2 (BatchNo  (None, 56, 56, 48)  192         ['conv2d_2[0][0]']               
 rmalization)                                                                                     
                                                                                                  
 activation_2 (Activation)      (None, 56, 56, 48)   0           ['batch_normalization_2[0][0]']  
                                                                                                  
 conv2d_3 (Conv2D)              (None, 56, 56, 48)   20784       ['activation_2[0][0]']           
                                                                                                  
 batch_normalization_3 (BatchNo  (None, 56, 56, 48)  192         ['conv2d_3[0][0]']               
 rmalization)                                                                                     
                                                                                                  
 activation_3 (Activation)      (None, 56, 56, 48)   0           ['batch_normalization_3[0][0]']  
                                                                                                  
 up_sampling2d_2 (UpSampling2D)  (None, 112, 112, 48  0          ['activation_3[0][0]']           
                                )                                                                 
                                                                                                  
 concatenate_2 (Concatenate)    (None, 112, 112, 96  0           ['up_sampling2d_2[0][0]',        
                                )                                 'block_1_expand_relu[0][0]']    
                                                                                                  
 conv2d_4 (Conv2D)              (None, 112, 112, 32  27680       ['concatenate_2[0][0]']          
                                )                                                                 
                                                                                                  
 batch_normalization_4 (BatchNo  (None, 112, 112, 32  128        ['conv2d_4[0][0]']               
 rmalization)                   )                                                                 
                                                                                                  
 activation_4 (Activation)      (None, 112, 112, 32  0           ['batch_normalization_4[0][0]']  
                                )                                                                 
                                                                                                  
 conv2d_5 (Conv2D)              (None, 112, 112, 32  9248        ['activation_4[0][0]']           
                                )                                                                 
                                                                                                  
 batch_normalization_5 (BatchNo  (None, 112, 112, 32  128        ['conv2d_5[0][0]']               
 rmalization)                   )                                                                 
                                                                                                  
 activation_5 (Activation)      (None, 112, 112, 32  0           ['batch_normalization_5[0][0]']  
                                )                                                                 
                                                                                                  
 up_sampling2d_3 (UpSampling2D)  (None, 224, 224, 32  0          ['activation_5[0][0]']           
                                )                                                                 
                                                                                                  
 concatenate_3 (Concatenate)    (None, 224, 224, 35  0           ['up_sampling2d_3[0][0]',        
                                )                                 'input_img[0][0]']              
                                                                                                  
 conv2d_6 (Conv2D)              (None, 224, 224, 16  5056        ['concatenate_3[0][0]']          
                                )                                                                 
                                                                                                  
 batch_normalization_6 (BatchNo  (None, 224, 224, 16  64         ['conv2d_6[0][0]']               
 rmalization)                   )                                                                 
                                                                                                  
 activation_6 (Activation)      (None, 224, 224, 16  0           ['batch_normalization_6[0][0]']  
                                )                                                                 
                                                                                                  
 conv2d_7 (Conv2D)              (None, 224, 224, 16  2320        ['activation_6[0][0]']           
                                )                                                                 
                                                                                                  
 batch_normalization_7 (BatchNo  (None, 224, 224, 16  64         ['conv2d_7[0][0]']               
 rmalization)                   )                                                                 
                                                                                                  
 activation_7 (Activation)      (None, 224, 224, 16  0           ['batch_normalization_7[0][0]']  
                                )                                                                 
                                                                                                  
 conv2d_8 (Conv2D)              (None, 224, 224, 1)  17          ['activation_7[0][0]']           
                                                                                                  
==================================================================================================
Total params: 568,369
Trainable params: 558,881
Non-trainable params: 9,488
__________________________________________________________________________________________________

Train the model

We will use Dice Coefficient for evaluation

In [ ]:
# define DICE-COEFFICIENT
smooth = tf.keras.backend.epsilon() # prevent zero error
def dice_coef(y_true, y_pred, smooth=1):
    """
    Dice = (2*|X & Y|)/ (|X|+ |Y|)
         =  2*sum(|A*B|)/(sum(A^2)+sum(B^2))
    ref: https://arxiv.org/pdf/1606.04797v1.pdf
    """
    intersection = tf.keras.backend.sum(tf.keras.backend.abs(y_true * y_pred), axis=-1)
    return (2. * intersection + smooth) / (tf.keras.backend.sum(tf.keras.backend.square(y_true),-1) + tf.keras.backend.sum(tf.keras.backend.square(y_pred),-1) + smooth)
In [ ]:
# LOSS CORRESPONDING TO DICE COEFF
def dice_coef_loss(y_true, y_pred):
    return 1-dice_coef(y_true, y_pred)
In [ ]:
# compile the model
opt = tf.keras.optimizers.Adam(lr=1e-4, beta_1=0.9, beta_2=0.999, epsilon=None, decay=0.0, amsgrad=False)
metrics = [dice_coef, tf.keras.metrics.Recall(), tf.keras.metrics.Precision()]
final_model.compile(loss=dice_coef_loss, optimizer=opt, metrics=metrics)
In [ ]:
BATCH_SIZE = 8
EPOCHS = 50
In [ ]:
# introduce model callbacks
callbacks = [
    tf.keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=4,min_lr=1e-6, verbose=1, mode="min"),
    tf.keras.callbacks.ModelCheckpoint("face_mask_model.h5", monitor="val_loss", verbose=1, save_best_only=True, mode="min")
]
In [ ]:
train_step_size = np.ceil(X_train.shape[0]/BATCH_SIZE)
val_step_size = np.ceil(X_val.shape[0]/BATCH_SIZE)
test_step_size = np.ceil(X_test.shape[0]/BATCH_SIZE)

train_step_size, val_step_size, test_step_size
Out[ ]:
(41.0, 9.0, 3.0)
In [ ]:
# train the model
training_history = final_model.fit(X_train, y_train,
          steps_per_epoch=train_step_size, 
          validation_data=(X_val,y_val), 
          validation_steps=val_step_size, 
          callbacks=callbacks,
          epochs=EPOCHS)
Epoch 1/50
41/41 [==============================] - ETA: 0s - loss: 0.4245 - dice_coef: 0.5755 - recall_2: 0.9045 - precision_2: 0.7315
Epoch 1: val_loss improved from inf to 0.41279, saving model to face_mask_model.h5
41/41 [==============================] - 17s 257ms/step - loss: 0.4245 - dice_coef: 0.5755 - recall_2: 0.9045 - precision_2: 0.7315 - val_loss: 0.4128 - val_dice_coef: 0.5872 - val_recall_2: 0.8882 - val_precision_2: 0.7163 - lr: 1.0000e-04
Epoch 2/50
41/41 [==============================] - ETA: 0s - loss: 0.4140 - dice_coef: 0.5860 - recall_2: 0.9155 - precision_2: 0.7332
Epoch 2: val_loss did not improve from 0.41279
41/41 [==============================] - 8s 192ms/step - loss: 0.4140 - dice_coef: 0.5860 - recall_2: 0.9155 - precision_2: 0.7332 - val_loss: 0.4134 - val_dice_coef: 0.5866 - val_recall_2: 0.8467 - val_precision_2: 0.7428 - lr: 1.0000e-04
Epoch 3/50
41/41 [==============================] - ETA: 0s - loss: 0.4039 - dice_coef: 0.5961 - recall_2: 0.9071 - precision_2: 0.7384
Epoch 3: val_loss improved from 0.41279 to 0.39813, saving model to face_mask_model.h5
41/41 [==============================] - 8s 207ms/step - loss: 0.4039 - dice_coef: 0.5961 - recall_2: 0.9071 - precision_2: 0.7384 - val_loss: 0.3981 - val_dice_coef: 0.6019 - val_recall_2: 0.8367 - val_precision_2: 0.7375 - lr: 1.0000e-04
Epoch 4/50
41/41 [==============================] - ETA: 0s - loss: 0.3940 - dice_coef: 0.6060 - recall_2: 0.9182 - precision_2: 0.7359
Epoch 4: val_loss did not improve from 0.39813
41/41 [==============================] - 8s 193ms/step - loss: 0.3940 - dice_coef: 0.6060 - recall_2: 0.9182 - precision_2: 0.7359 - val_loss: 0.4013 - val_dice_coef: 0.5987 - val_recall_2: 0.8680 - val_precision_2: 0.6750 - lr: 1.0000e-04
Epoch 5/50
41/41 [==============================] - ETA: 0s - loss: 0.3872 - dice_coef: 0.6128 - recall_2: 0.9130 - precision_2: 0.7458
Epoch 5: val_loss improved from 0.39813 to 0.38428, saving model to face_mask_model.h5
41/41 [==============================] - 9s 208ms/step - loss: 0.3872 - dice_coef: 0.6128 - recall_2: 0.9130 - precision_2: 0.7458 - val_loss: 0.3843 - val_dice_coef: 0.6157 - val_recall_2: 0.8229 - val_precision_2: 0.7597 - lr: 1.0000e-04
Epoch 6/50
41/41 [==============================] - ETA: 0s - loss: 0.3777 - dice_coef: 0.6223 - recall_2: 0.9090 - precision_2: 0.7571
Epoch 6: val_loss improved from 0.38428 to 0.38278, saving model to face_mask_model.h5
41/41 [==============================] - 9s 208ms/step - loss: 0.3777 - dice_coef: 0.6223 - recall_2: 0.9090 - precision_2: 0.7571 - val_loss: 0.3828 - val_dice_coef: 0.6172 - val_recall_2: 0.7998 - val_precision_2: 0.7929 - lr: 1.0000e-04
Epoch 7/50
41/41 [==============================] - ETA: 0s - loss: 0.3654 - dice_coef: 0.6346 - recall_2: 0.9117 - precision_2: 0.7412
Epoch 7: val_loss did not improve from 0.38278
41/41 [==============================] - 8s 192ms/step - loss: 0.3654 - dice_coef: 0.6346 - recall_2: 0.9117 - precision_2: 0.7412 - val_loss: 0.3883 - val_dice_coef: 0.6117 - val_recall_2: 0.7259 - val_precision_2: 0.8162 - lr: 1.0000e-04
Epoch 8/50
41/41 [==============================] - ETA: 0s - loss: 0.3565 - dice_coef: 0.6435 - recall_2: 0.9137 - precision_2: 0.7637
Epoch 8: val_loss improved from 0.38278 to 0.38203, saving model to face_mask_model.h5
41/41 [==============================] - 9s 209ms/step - loss: 0.3565 - dice_coef: 0.6435 - recall_2: 0.9137 - precision_2: 0.7637 - val_loss: 0.3820 - val_dice_coef: 0.6180 - val_recall_2: 0.8400 - val_precision_2: 0.7631 - lr: 1.0000e-04
Epoch 9/50
41/41 [==============================] - ETA: 0s - loss: 0.3511 - dice_coef: 0.6489 - recall_2: 0.9057 - precision_2: 0.7575
Epoch 9: val_loss did not improve from 0.38203
41/41 [==============================] - 8s 196ms/step - loss: 0.3511 - dice_coef: 0.6489 - recall_2: 0.9057 - precision_2: 0.7575 - val_loss: 0.3874 - val_dice_coef: 0.6126 - val_recall_2: 0.8648 - val_precision_2: 0.6646 - lr: 1.0000e-04
Epoch 10/50
41/41 [==============================] - ETA: 0s - loss: 0.3433 - dice_coef: 0.6567 - recall_2: 0.9155 - precision_2: 0.7783
Epoch 10: val_loss improved from 0.38203 to 0.37342, saving model to face_mask_model.h5
41/41 [==============================] - 9s 213ms/step - loss: 0.3433 - dice_coef: 0.6567 - recall_2: 0.9155 - precision_2: 0.7783 - val_loss: 0.3734 - val_dice_coef: 0.6266 - val_recall_2: 0.6897 - val_precision_2: 0.8524 - lr: 1.0000e-04
Epoch 11/50
41/41 [==============================] - ETA: 0s - loss: 0.3372 - dice_coef: 0.6628 - recall_2: 0.9030 - precision_2: 0.7811
Epoch 11: val_loss did not improve from 0.37342
41/41 [==============================] - 8s 191ms/step - loss: 0.3372 - dice_coef: 0.6628 - recall_2: 0.9030 - precision_2: 0.7811 - val_loss: 0.3779 - val_dice_coef: 0.6221 - val_recall_2: 0.7979 - val_precision_2: 0.7681 - lr: 1.0000e-04
Epoch 12/50
41/41 [==============================] - ETA: 0s - loss: 0.3287 - dice_coef: 0.6713 - recall_2: 0.9058 - precision_2: 0.7696
Epoch 12: val_loss did not improve from 0.37342
41/41 [==============================] - 8s 190ms/step - loss: 0.3287 - dice_coef: 0.6713 - recall_2: 0.9058 - precision_2: 0.7696 - val_loss: 0.3860 - val_dice_coef: 0.6140 - val_recall_2: 0.9052 - val_precision_2: 0.6342 - lr: 1.0000e-04
Epoch 13/50
41/41 [==============================] - ETA: 0s - loss: 0.3209 - dice_coef: 0.6791 - recall_2: 0.9059 - precision_2: 0.7748
Epoch 13: val_loss improved from 0.37342 to 0.37025, saving model to face_mask_model.h5
41/41 [==============================] - 9s 208ms/step - loss: 0.3209 - dice_coef: 0.6791 - recall_2: 0.9059 - precision_2: 0.7748 - val_loss: 0.3702 - val_dice_coef: 0.6298 - val_recall_2: 0.8239 - val_precision_2: 0.7499 - lr: 1.0000e-04
Epoch 14/50
41/41 [==============================] - ETA: 0s - loss: 0.3127 - dice_coef: 0.6873 - recall_2: 0.9085 - precision_2: 0.7704
Epoch 14: val_loss improved from 0.37025 to 0.34883, saving model to face_mask_model.h5
41/41 [==============================] - 9s 209ms/step - loss: 0.3127 - dice_coef: 0.6873 - recall_2: 0.9085 - precision_2: 0.7704 - val_loss: 0.3488 - val_dice_coef: 0.6512 - val_recall_2: 0.8551 - val_precision_2: 0.6656 - lr: 1.0000e-04
Epoch 15/50
41/41 [==============================] - ETA: 0s - loss: 0.3047 - dice_coef: 0.6953 - recall_2: 0.9055 - precision_2: 0.7729
Epoch 15: val_loss did not improve from 0.34883
41/41 [==============================] - 8s 194ms/step - loss: 0.3047 - dice_coef: 0.6953 - recall_2: 0.9055 - precision_2: 0.7729 - val_loss: 0.3709 - val_dice_coef: 0.6291 - val_recall_2: 0.8039 - val_precision_2: 0.7739 - lr: 1.0000e-04
Epoch 16/50
41/41 [==============================] - ETA: 0s - loss: 0.2973 - dice_coef: 0.7027 - recall_2: 0.9114 - precision_2: 0.7920
Epoch 16: val_loss did not improve from 0.34883
41/41 [==============================] - 8s 192ms/step - loss: 0.2973 - dice_coef: 0.7027 - recall_2: 0.9114 - precision_2: 0.7920 - val_loss: 0.3553 - val_dice_coef: 0.6447 - val_recall_2: 0.8065 - val_precision_2: 0.7542 - lr: 1.0000e-04
Epoch 17/50
41/41 [==============================] - ETA: 0s - loss: 0.2911 - dice_coef: 0.7089 - recall_2: 0.9131 - precision_2: 0.7835
Epoch 17: val_loss did not improve from 0.34883
41/41 [==============================] - 8s 191ms/step - loss: 0.2911 - dice_coef: 0.7089 - recall_2: 0.9131 - precision_2: 0.7835 - val_loss: 0.3682 - val_dice_coef: 0.6318 - val_recall_2: 0.8235 - val_precision_2: 0.7259 - lr: 1.0000e-04
Epoch 18/50
41/41 [==============================] - ETA: 0s - loss: 0.2846 - dice_coef: 0.7154 - recall_2: 0.9106 - precision_2: 0.7911
Epoch 18: ReduceLROnPlateau reducing learning rate to 9.999999747378752e-06.

Epoch 18: val_loss did not improve from 0.34883
41/41 [==============================] - 8s 192ms/step - loss: 0.2846 - dice_coef: 0.7154 - recall_2: 0.9106 - precision_2: 0.7911 - val_loss: 0.3798 - val_dice_coef: 0.6202 - val_recall_2: 0.8191 - val_precision_2: 0.7101 - lr: 1.0000e-04
Epoch 19/50
41/41 [==============================] - ETA: 0s - loss: 0.2802 - dice_coef: 0.7198 - recall_2: 0.9036 - precision_2: 0.8102
Epoch 19: val_loss did not improve from 0.34883
41/41 [==============================] - 8s 192ms/step - loss: 0.2802 - dice_coef: 0.7198 - recall_2: 0.9036 - precision_2: 0.8102 - val_loss: 0.3672 - val_dice_coef: 0.6328 - val_recall_2: 0.8762 - val_precision_2: 0.6852 - lr: 1.0000e-05
Epoch 20/50
41/41 [==============================] - ETA: 0s - loss: 0.2788 - dice_coef: 0.7212 - recall_2: 0.9167 - precision_2: 0.8106
Epoch 20: val_loss did not improve from 0.34883
41/41 [==============================] - 8s 192ms/step - loss: 0.2788 - dice_coef: 0.7212 - recall_2: 0.9167 - precision_2: 0.8106 - val_loss: 0.3584 - val_dice_coef: 0.6416 - val_recall_2: 0.8622 - val_precision_2: 0.7000 - lr: 1.0000e-05
Epoch 21/50
41/41 [==============================] - ETA: 0s - loss: 0.2797 - dice_coef: 0.7203 - recall_2: 0.9112 - precision_2: 0.7945
Epoch 21: val_loss did not improve from 0.34883
41/41 [==============================] - 8s 196ms/step - loss: 0.2797 - dice_coef: 0.7203 - recall_2: 0.9112 - precision_2: 0.7945 - val_loss: 0.3567 - val_dice_coef: 0.6433 - val_recall_2: 0.8608 - val_precision_2: 0.7072 - lr: 1.0000e-05
Epoch 22/50
41/41 [==============================] - ETA: 0s - loss: 0.2768 - dice_coef: 0.7232 - recall_2: 0.9162 - precision_2: 0.8030
Epoch 22: ReduceLROnPlateau reducing learning rate to 1e-06.

Epoch 22: val_loss did not improve from 0.34883
41/41 [==============================] - 8s 193ms/step - loss: 0.2768 - dice_coef: 0.7232 - recall_2: 0.9162 - precision_2: 0.8030 - val_loss: 0.3557 - val_dice_coef: 0.6443 - val_recall_2: 0.8732 - val_precision_2: 0.6960 - lr: 1.0000e-05
Epoch 23/50
41/41 [==============================] - ETA: 0s - loss: 0.2764 - dice_coef: 0.7236 - recall_2: 0.9142 - precision_2: 0.8054
Epoch 23: val_loss did not improve from 0.34883
41/41 [==============================] - 8s 192ms/step - loss: 0.2764 - dice_coef: 0.7236 - recall_2: 0.9142 - precision_2: 0.8054 - val_loss: 0.3534 - val_dice_coef: 0.6466 - val_recall_2: 0.8723 - val_precision_2: 0.6991 - lr: 1.0000e-06
Epoch 24/50
41/41 [==============================] - ETA: 0s - loss: 0.2763 - dice_coef: 0.7237 - recall_2: 0.9138 - precision_2: 0.8018
Epoch 24: val_loss did not improve from 0.34883
41/41 [==============================] - 8s 191ms/step - loss: 0.2763 - dice_coef: 0.7237 - recall_2: 0.9138 - precision_2: 0.8018 - val_loss: 0.3514 - val_dice_coef: 0.6486 - val_recall_2: 0.8705 - val_precision_2: 0.7028 - lr: 1.0000e-06
Epoch 25/50
41/41 [==============================] - ETA: 0s - loss: 0.2763 - dice_coef: 0.7237 - recall_2: 0.9178 - precision_2: 0.7986
Epoch 25: val_loss did not improve from 0.34883
41/41 [==============================] - 8s 191ms/step - loss: 0.2763 - dice_coef: 0.7237 - recall_2: 0.9178 - precision_2: 0.7986 - val_loss: 0.3496 - val_dice_coef: 0.6504 - val_recall_2: 0.8693 - val_precision_2: 0.7064 - lr: 1.0000e-06
Epoch 26/50
41/41 [==============================] - ETA: 0s - loss: 0.2772 - dice_coef: 0.7228 - recall_2: 0.9108 - precision_2: 0.7932
Epoch 26: val_loss improved from 0.34883 to 0.34802, saving model to face_mask_model.h5
41/41 [==============================] - 9s 209ms/step - loss: 0.2772 - dice_coef: 0.7228 - recall_2: 0.9108 - precision_2: 0.7932 - val_loss: 0.3480 - val_dice_coef: 0.6520 - val_recall_2: 0.8694 - val_precision_2: 0.7074 - lr: 1.0000e-06
Epoch 27/50
41/41 [==============================] - ETA: 0s - loss: 0.2742 - dice_coef: 0.7258 - recall_2: 0.9208 - precision_2: 0.8014
Epoch 27: val_loss improved from 0.34802 to 0.34616, saving model to face_mask_model.h5
41/41 [==============================] - 9s 209ms/step - loss: 0.2742 - dice_coef: 0.7258 - recall_2: 0.9208 - precision_2: 0.8014 - val_loss: 0.3462 - val_dice_coef: 0.6538 - val_recall_2: 0.8697 - val_precision_2: 0.7076 - lr: 1.0000e-06
Epoch 28/50
41/41 [==============================] - ETA: 0s - loss: 0.2751 - dice_coef: 0.7249 - recall_2: 0.9190 - precision_2: 0.7984
Epoch 28: val_loss improved from 0.34616 to 0.34450, saving model to face_mask_model.h5
41/41 [==============================] - 9s 209ms/step - loss: 0.2751 - dice_coef: 0.7249 - recall_2: 0.9190 - precision_2: 0.7984 - val_loss: 0.3445 - val_dice_coef: 0.6555 - val_recall_2: 0.8694 - val_precision_2: 0.7089 - lr: 1.0000e-06
Epoch 29/50
41/41 [==============================] - ETA: 0s - loss: 0.2750 - dice_coef: 0.7250 - recall_2: 0.9211 - precision_2: 0.7975
Epoch 29: val_loss improved from 0.34450 to 0.34335, saving model to face_mask_model.h5
41/41 [==============================] - 9s 210ms/step - loss: 0.2750 - dice_coef: 0.7250 - recall_2: 0.9211 - precision_2: 0.7975 - val_loss: 0.3433 - val_dice_coef: 0.6567 - val_recall_2: 0.8695 - val_precision_2: 0.7092 - lr: 1.0000e-06
Epoch 30/50
41/41 [==============================] - ETA: 0s - loss: 0.2791 - dice_coef: 0.7209 - recall_2: 0.9067 - precision_2: 0.7885
Epoch 30: val_loss improved from 0.34335 to 0.34238, saving model to face_mask_model.h5
41/41 [==============================] - 9s 209ms/step - loss: 0.2791 - dice_coef: 0.7209 - recall_2: 0.9067 - precision_2: 0.7885 - val_loss: 0.3424 - val_dice_coef: 0.6576 - val_recall_2: 0.8693 - val_precision_2: 0.7108 - lr: 1.0000e-06
Epoch 31/50
41/41 [==============================] - ETA: 0s - loss: 0.2741 - dice_coef: 0.7259 - recall_2: 0.9199 - precision_2: 0.8028
Epoch 31: val_loss improved from 0.34238 to 0.34092, saving model to face_mask_model.h5
41/41 [==============================] - 9s 213ms/step - loss: 0.2741 - dice_coef: 0.7259 - recall_2: 0.9199 - precision_2: 0.8028 - val_loss: 0.3409 - val_dice_coef: 0.6591 - val_recall_2: 0.8682 - val_precision_2: 0.7144 - lr: 1.0000e-06
Epoch 32/50
41/41 [==============================] - ETA: 0s - loss: 0.2764 - dice_coef: 0.7236 - recall_2: 0.9140 - precision_2: 0.7942
Epoch 32: val_loss improved from 0.34092 to 0.33968, saving model to face_mask_model.h5
41/41 [==============================] - 9s 209ms/step - loss: 0.2764 - dice_coef: 0.7236 - recall_2: 0.9140 - precision_2: 0.7942 - val_loss: 0.3397 - val_dice_coef: 0.6603 - val_recall_2: 0.8673 - val_precision_2: 0.7167 - lr: 1.0000e-06
Epoch 33/50
41/41 [==============================] - ETA: 0s - loss: 0.2747 - dice_coef: 0.7253 - recall_2: 0.9229 - precision_2: 0.8035
Epoch 33: val_loss improved from 0.33968 to 0.33854, saving model to face_mask_model.h5
41/41 [==============================] - 9s 210ms/step - loss: 0.2747 - dice_coef: 0.7253 - recall_2: 0.9229 - precision_2: 0.8035 - val_loss: 0.3385 - val_dice_coef: 0.6615 - val_recall_2: 0.8670 - val_precision_2: 0.7183 - lr: 1.0000e-06
Epoch 34/50
41/41 [==============================] - ETA: 0s - loss: 0.2773 - dice_coef: 0.7227 - recall_2: 0.9118 - precision_2: 0.7950
Epoch 34: val_loss improved from 0.33854 to 0.33762, saving model to face_mask_model.h5
41/41 [==============================] - 9s 209ms/step - loss: 0.2773 - dice_coef: 0.7227 - recall_2: 0.9118 - precision_2: 0.7950 - val_loss: 0.3376 - val_dice_coef: 0.6624 - val_recall_2: 0.8651 - val_precision_2: 0.7213 - lr: 1.0000e-06
Epoch 35/50
41/41 [==============================] - ETA: 0s - loss: 0.2762 - dice_coef: 0.7238 - recall_2: 0.9164 - precision_2: 0.8041
Epoch 35: val_loss improved from 0.33762 to 0.33684, saving model to face_mask_model.h5
41/41 [==============================] - 9s 209ms/step - loss: 0.2762 - dice_coef: 0.7238 - recall_2: 0.9164 - precision_2: 0.8041 - val_loss: 0.3368 - val_dice_coef: 0.6632 - val_recall_2: 0.8655 - val_precision_2: 0.7225 - lr: 1.0000e-06
Epoch 36/50
41/41 [==============================] - ETA: 0s - loss: 0.2755 - dice_coef: 0.7245 - recall_2: 0.9206 - precision_2: 0.8025
Epoch 36: val_loss improved from 0.33684 to 0.33564, saving model to face_mask_model.h5
41/41 [==============================] - 9s 209ms/step - loss: 0.2755 - dice_coef: 0.7245 - recall_2: 0.9206 - precision_2: 0.8025 - val_loss: 0.3356 - val_dice_coef: 0.6644 - val_recall_2: 0.8646 - val_precision_2: 0.7251 - lr: 1.0000e-06
Epoch 37/50
41/41 [==============================] - ETA: 0s - loss: 0.2766 - dice_coef: 0.7234 - recall_2: 0.9102 - precision_2: 0.7955
Epoch 37: val_loss improved from 0.33564 to 0.33454, saving model to face_mask_model.h5
41/41 [==============================] - 9s 209ms/step - loss: 0.2766 - dice_coef: 0.7234 - recall_2: 0.9102 - precision_2: 0.7955 - val_loss: 0.3345 - val_dice_coef: 0.6655 - val_recall_2: 0.8640 - val_precision_2: 0.7267 - lr: 1.0000e-06
Epoch 38/50
41/41 [==============================] - ETA: 0s - loss: 0.2746 - dice_coef: 0.7254 - recall_2: 0.9190 - precision_2: 0.7963
Epoch 38: val_loss improved from 0.33454 to 0.33323, saving model to face_mask_model.h5
41/41 [==============================] - 9s 210ms/step - loss: 0.2746 - dice_coef: 0.7254 - recall_2: 0.9190 - precision_2: 0.7963 - val_loss: 0.3332 - val_dice_coef: 0.6668 - val_recall_2: 0.8623 - val_precision_2: 0.7281 - lr: 1.0000e-06
Epoch 39/50
41/41 [==============================] - ETA: 0s - loss: 0.2752 - dice_coef: 0.7248 - recall_2: 0.9179 - precision_2: 0.7998
Epoch 39: val_loss improved from 0.33323 to 0.33290, saving model to face_mask_model.h5
41/41 [==============================] - 9s 210ms/step - loss: 0.2752 - dice_coef: 0.7248 - recall_2: 0.9179 - precision_2: 0.7998 - val_loss: 0.3329 - val_dice_coef: 0.6671 - val_recall_2: 0.8620 - val_precision_2: 0.7295 - lr: 1.0000e-06
Epoch 40/50
41/41 [==============================] - ETA: 0s - loss: 0.2751 - dice_coef: 0.7249 - recall_2: 0.9187 - precision_2: 0.7972
Epoch 40: val_loss improved from 0.33290 to 0.33220, saving model to face_mask_model.h5
41/41 [==============================] - 9s 215ms/step - loss: 0.2751 - dice_coef: 0.7249 - recall_2: 0.9187 - precision_2: 0.7972 - val_loss: 0.3322 - val_dice_coef: 0.6678 - val_recall_2: 0.8613 - val_precision_2: 0.7313 - lr: 1.0000e-06
Epoch 41/50
41/41 [==============================] - ETA: 0s - loss: 0.2754 - dice_coef: 0.7246 - recall_2: 0.9177 - precision_2: 0.7947
Epoch 41: val_loss improved from 0.33220 to 0.33106, saving model to face_mask_model.h5
41/41 [==============================] - 9s 214ms/step - loss: 0.2754 - dice_coef: 0.7246 - recall_2: 0.9177 - precision_2: 0.7947 - val_loss: 0.3311 - val_dice_coef: 0.6689 - val_recall_2: 0.8607 - val_precision_2: 0.7330 - lr: 1.0000e-06
Epoch 42/50
41/41 [==============================] - ETA: 0s - loss: 0.2754 - dice_coef: 0.7246 - recall_2: 0.9181 - precision_2: 0.7960
Epoch 42: val_loss improved from 0.33106 to 0.33041, saving model to face_mask_model.h5
41/41 [==============================] - 9s 214ms/step - loss: 0.2754 - dice_coef: 0.7246 - recall_2: 0.9181 - precision_2: 0.7960 - val_loss: 0.3304 - val_dice_coef: 0.6696 - val_recall_2: 0.8582 - val_precision_2: 0.7362 - lr: 1.0000e-06
Epoch 43/50
41/41 [==============================] - ETA: 0s - loss: 0.2736 - dice_coef: 0.7264 - recall_2: 0.9215 - precision_2: 0.8035
Epoch 43: val_loss improved from 0.33041 to 0.32960, saving model to face_mask_model.h5
41/41 [==============================] - 9s 210ms/step - loss: 0.2736 - dice_coef: 0.7264 - recall_2: 0.9215 - precision_2: 0.8035 - val_loss: 0.3296 - val_dice_coef: 0.6704 - val_recall_2: 0.8582 - val_precision_2: 0.7374 - lr: 1.0000e-06
Epoch 44/50
41/41 [==============================] - ETA: 0s - loss: 0.2757 - dice_coef: 0.7243 - recall_2: 0.9147 - precision_2: 0.7962
Epoch 44: val_loss improved from 0.32960 to 0.32885, saving model to face_mask_model.h5
41/41 [==============================] - 9s 209ms/step - loss: 0.2757 - dice_coef: 0.7243 - recall_2: 0.9147 - precision_2: 0.7962 - val_loss: 0.3289 - val_dice_coef: 0.6711 - val_recall_2: 0.8578 - val_precision_2: 0.7375 - lr: 1.0000e-06
Epoch 45/50
41/41 [==============================] - ETA: 0s - loss: 0.2746 - dice_coef: 0.7254 - recall_2: 0.9185 - precision_2: 0.7946
Epoch 45: val_loss improved from 0.32885 to 0.32859, saving model to face_mask_model.h5
41/41 [==============================] - 9s 214ms/step - loss: 0.2746 - dice_coef: 0.7254 - recall_2: 0.9185 - precision_2: 0.7946 - val_loss: 0.3286 - val_dice_coef: 0.6714 - val_recall_2: 0.8586 - val_precision_2: 0.7373 - lr: 1.0000e-06
Epoch 46/50
41/41 [==============================] - ETA: 0s - loss: 0.2734 - dice_coef: 0.7266 - recall_2: 0.9239 - precision_2: 0.8063
Epoch 46: val_loss improved from 0.32859 to 0.32815, saving model to face_mask_model.h5
41/41 [==============================] - 9s 210ms/step - loss: 0.2734 - dice_coef: 0.7266 - recall_2: 0.9239 - precision_2: 0.8063 - val_loss: 0.3281 - val_dice_coef: 0.6719 - val_recall_2: 0.8578 - val_precision_2: 0.7390 - lr: 1.0000e-06
Epoch 47/50
41/41 [==============================] - ETA: 0s - loss: 0.2749 - dice_coef: 0.7251 - recall_2: 0.9188 - precision_2: 0.8024
Epoch 47: val_loss improved from 0.32815 to 0.32744, saving model to face_mask_model.h5
41/41 [==============================] - 9s 209ms/step - loss: 0.2749 - dice_coef: 0.7251 - recall_2: 0.9188 - precision_2: 0.8024 - val_loss: 0.3274 - val_dice_coef: 0.6726 - val_recall_2: 0.8565 - val_precision_2: 0.7412 - lr: 1.0000e-06
Epoch 48/50
41/41 [==============================] - ETA: 0s - loss: 0.2741 - dice_coef: 0.7259 - recall_2: 0.9213 - precision_2: 0.8019
Epoch 48: val_loss improved from 0.32744 to 0.32665, saving model to face_mask_model.h5
41/41 [==============================] - 9s 211ms/step - loss: 0.2741 - dice_coef: 0.7259 - recall_2: 0.9213 - precision_2: 0.8019 - val_loss: 0.3267 - val_dice_coef: 0.6733 - val_recall_2: 0.8561 - val_precision_2: 0.7417 - lr: 1.0000e-06
Epoch 49/50
41/41 [==============================] - ETA: 0s - loss: 0.2755 - dice_coef: 0.7245 - recall_2: 0.9156 - precision_2: 0.7926
Epoch 49: val_loss improved from 0.32665 to 0.32625, saving model to face_mask_model.h5
41/41 [==============================] - 9s 213ms/step - loss: 0.2755 - dice_coef: 0.7245 - recall_2: 0.9156 - precision_2: 0.7926 - val_loss: 0.3263 - val_dice_coef: 0.6737 - val_recall_2: 0.8553 - val_precision_2: 0.7427 - lr: 1.0000e-06
Epoch 50/50
41/41 [==============================] - ETA: 0s - loss: 0.2743 - dice_coef: 0.7257 - recall_2: 0.9201 - precision_2: 0.8023
Epoch 50: val_loss improved from 0.32625 to 0.32586, saving model to face_mask_model.h5
41/41 [==============================] - 9s 210ms/step - loss: 0.2743 - dice_coef: 0.7257 - recall_2: 0.9201 - precision_2: 0.8023 - val_loss: 0.3259 - val_dice_coef: 0.6741 - val_recall_2: 0.8549 - val_precision_2: 0.7445 - lr: 1.0000e-06
In [ ]:
def plot_train_history_with_epochs(training_history):
  '''
  This function is used to plot Training and Validation loss
  versus the number of epochs based on the training history of the model.
  '''
  # extracting training and validation losses from model history
  train_loss = training_history.history['loss']
  val_loss = training_history.history['val_loss']
  
  # number of epochs
  epochs = range(len(train_loss))

  # plot train and validation loss w.r.t number of epochs
  plt.figure(figsize=(10,8))
  plt.plot(epochs, train_loss, label = 'training loss')
  plt.plot(epochs, val_loss, label = 'validation loss')
  plt.legend(loc='upper right')
  plt.xlabel('Number of Epochs')
  plt.ylabel('Training and Validation loss')
  plt.title('Training and Validation loss v/s Epochs')
  plt.show()
In [ ]:
# load best saved model
loaded_model = tf.keras.models.load_model('face_mask_model.h5',
                                          custom_objects={'dice_coef_loss':dice_coef_loss,'dice_coef':dice_coef})
In [ ]:
# evaluating the loaded model on train data
train_metrics = loaded_model.evaluate(X_train,y_train,verbose=0,steps=train_step_size)
In [ ]:
# evaluating the loaded model on validation data
val_metrics = loaded_model.evaluate(X_val,y_val,verbose=0,steps=val_step_size)
In [ ]:
# evaluating the loaded model on test data
test_metrics = loaded_model.evaluate(X_test,y_test,verbose=0,steps=test_step_size)
In [ ]:
# displaying the evaluated metrics
metrics_df = pd.DataFrame(index=['Loss','Dice Coefficient', 'Recall','Precision'],columns=['Training','Validation','Testing'],
                          data=[[train_metrics[0],val_metrics[0],test_metrics[0]],
                                [train_metrics[1],val_metrics[1],test_metrics[1]],
                                [train_metrics[2],val_metrics[2],test_metrics[2]],
                                [train_metrics[3],val_metrics[3],test_metrics[3]]])
metrics_df
Out[ ]:
Training Validation Testing
Loss 0.274687 0.325858 0.346595
Dice Coefficient 0.725313 0.674142 0.653405
Recall 0.962953 0.854891 0.865303
Precision 0.785426 0.744479 0.730708
In [ ]:
# call to appropriate function for train history
plot_train_history_with_epochs(training_history)

Observations:

  • The model was optimized using Adam optimizer.
  • The loss function was custom loss function of Dice Coefficient which deals with overlap of masks with detected area.
  • The model was evaluated on metrics like dice coefficient, precision and recall.
  • The model trained smoothly with decreasing loss.
  • Also the metrics across train, validation and test seem to be good.

Predicting using the saved model

In [ ]:
# copy prediction image
!cp '/content/drive/MyDrive/Colab Notebooks/Part 1Test Data - Prediction Image.jpeg' .
In [ ]:
# verify copying
!ls -l
total 1332532
drwx------ 5 root root       4096 Mar 20 12:43  drive
-rw-r--r-- 1 root root    7575920 Mar 20 14:19  face_mask_model.h5
-rw------- 1 root root 1356868279 Mar 20 12:44  images.npy
-rw------- 1 root root      52325 Mar 20 14:27 'Part 1Test Data - Prediction Image.jpeg'
drwxr-xr-x 1 root root       4096 Mar  9 14:48  sample_data
In [ ]:
# load the prediction image
file_path = '/content/Part 1Test Data - Prediction Image.jpeg'
img = cv2.imread(file_path)
#Resize image
img = cv2.resize(img, dsize= (IMG_WIDTH,IMG_HEIGHT), interpolation=cv2.INTER_CUBIC)
# convert to RGB
img = cv2.cvtColor(img,cv2.COLOR_BGR2RGB)
# pre-process as per the model
processed_img = tf.keras.applications.mobilenet_v2.preprocess_input(np.array(img, dtype=np.float32))

# expanding dimensions into tensor
processed_img = np.expand_dims(processed_img,axis=0)
In [ ]:
processed_img.shape
Out[ ]:
(1, 224, 224, 3)
In [ ]:
# predicting using the best saved model
y_pred = loaded_model.predict(processed_img)
In [ ]:
y_pred.shape
Out[ ]:
(1, 224, 224, 1)
In [ ]:
# assigning label beyond a threshold and resize
pred_mask = cv2.resize((1.0*(y_pred[0] > 0.7)), (IMG_WIDTH,IMG_HEIGHT))
In [ ]:
# visualise masks and images for prediction image
# initialising subplots
figure, ax = plt.subplots(nrows=1, ncols=2)

# setting figure parameters
figure.set_figheight(15)
figure.set_figwidth(15)

# setting images and masks to axis
ax[0].imshow(processed_img[0])
ax[0].axis('off')
ax[1].imshow(pred_mask)
ax[1].axis('off')

plt.show()
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers).

Observations:

  • The model mask predictions for the given image seem to be fairly good.
  • The model can be tuned further for more better results.

Project Two - Face Recognition

  • DOMAIN: Face recognition
  • CONTEXT: Company X intends to build a face identification model to recognise human faces.
  • DATA DESCRIPTION: The dataset comprises of images and its mask where there is a human face.
  • PROJECT OBJECTIVE: Face Aligned Face Dataset from Pinterest. This dataset contains 10,770 images for 100 people. All images are taken from Pinterest and aligned using dlib library.

Import libraries

Most all the libraries are imported from above. We will be re-using them.

In [ ]:
# operating system library
import os

# for visualising images in a grid
from mpl_toolkits.axes_grid1 import ImageGrid

# for pre-processing - encode labels, scaling
from sklearn.preprocessing import LabelEncoder,StandardScaler

# searching for hyperparameter tuning
from sklearn.model_selection import GridSearchCV

# for dimensionality reduction using Principal Component Analysis
from sklearn.decomposition import PCA

# ML model libraries
from sklearn import svm, metrics

Load the dataset

In [ ]:
# copy the content to current directory
!cp '/content/drive/MyDrive/Colab Notebooks/Aligned+Face+Dataset+from+Pinterest+-+CV+project+1.zip' .
In [ ]:
# verify copying
!ls -l
total 467548
-rw------- 1 root root 478756252 Mar 26 12:18 Aligned+Face+Dataset+from+Pinterest+-+CV+project+1.zip
drwx------ 5 root root      4096 Mar 26 12:16 drive
drwxr-xr-x 1 root root      4096 Mar 23 14:22 sample_data
In [ ]:
# unzip the data either using command or zipfile module
!unzip /content/Aligned+Face+Dataset+from+Pinterest+-+CV+project+1.zip
In [ ]:
# verify after unzipping the content
!ls -l
total 467556
-rw-------   1 root root 478756252 Mar 26 12:18 Aligned+Face+Dataset+from+Pinterest+-+CV+project+1.zip
drwx------   5 root root      4096 Mar 26 12:16 drive
drwxr-xr-x   3 root root      4096 Mar 26 12:18 pins
drwxr-xr-x 102 root root      4096 Mar 26 12:18 PINS
drwxr-xr-x   1 root root      4096 Mar 23 14:22 sample_data
In [ ]:
# Observing data structure
!ls -l pins/
total 4
drwxr-xr-x 102 root root 4096 Mar 26 12:18 PINS
In [ ]:
!ls -l pins/PINS
total 568
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Aaron Paul'
drwxr-xr-x 2 root root 12288 Mar 26 12:18 'pins_alexandra daddario'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Alvaro Morte'
drwxr-xr-x 2 root root 12288 Mar 26 12:18 'pins_alycia debnam carey face'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Amanda Crew'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Amaury Nolasco'
drwxr-xr-x 2 root root 12288 Mar 26 12:18 'pins_amber heard face'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Anna Gunn'
drwxr-xr-x 2 root root 12288 Mar 26 12:18 'pins_anne hathaway'
drwxr-xr-x 2 root root 12288 Mar 26 12:18 'pins_barbara palvin face'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_bellamy blake face'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Benedict Cumberbatch'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Betsy Brandt'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_bill gates'
drwxr-xr-x 2 root root 12288 Mar 26 12:18 'pins_Brenton Thwaites'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_brie larson'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Brit Marling'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Bryan Cranston'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Caity Lotz'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Cameron Monaghan'
drwxr-xr-x 2 root root 12288 Mar 26 12:18 'pins_chadwick boseman face'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Chance Perdomo'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Chris Evans'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Chris Pratt'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Cobie Smulders'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Danielle Panabaker'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Dave Franco'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_david mazouz'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Dominic Purcell'
drwxr-xr-x 2 root root  4096 Mar 26 12:18  pins_drake
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_dua lipa face'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Dwayne Johnson'
drwxr-xr-x 2 root root 12288 Mar 26 12:18 'pins_elizabeth olsen face'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_eliza taylor'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_elon musk'
drwxr-xr-x 2 root root 12288 Mar 26 12:18 'pins_Emilia Clarke'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Emily Bett Rickards'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Emma Stone'
drwxr-xr-x 2 root root 12288 Mar 26 12:18 'pins_emma watson face'
drwxr-xr-x 2 root root 12288 Mar 26 12:18 'pins_gal gadot face'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_grant gustin face'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Gwyneth Paltrow'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Henry Cavil'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_jason isaacs'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Jason Momoa'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_jeff bezos'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Jeremy Renner'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Jesse Eisenberg'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Jim Parsons'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Jon Bernthal'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Josh Radnor'
drwxr-xr-x 2 root root 12288 Mar 26 12:18 'pins_kiernan shipka '
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Kit Harington'
drwxr-xr-x 2 root root 12288 Mar 26 12:18 'pins_kristen stewart face'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Krysten Ritter'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Kumail Nanjiani'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_lindsey morgan face'
drwxr-xr-x 2 root root 12288 Mar 26 12:18 'pins_Maisie Williams'
drwxr-xr-x 2 root root 12288 Mar 26 12:18 'pins_margot robbie face'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_maria pedraza'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Mark Ruffalo'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_mark zuckerberg'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Martin Starr'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Melissa benoit'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_miguel herran'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Mike Colter'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_millie bobby brown'
drwxr-xr-x 2 root root 12288 Mar 26 12:18 'pins_Morena Baccarin'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Morgan Freeman'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Natalie Portman '
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Neil Patrick Harris'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Paul Rudd'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Pedro Alonso'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Peter Dinklage'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Rami Melek'
drwxr-xr-x 2 root root  4096 Mar 26 12:18  pins_rihanna
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_RJ Mitte'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_robert downey jr face'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Robert Knepper'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Robin Taylor'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Ryan Reynolds'
drwxr-xr-x 2 root root 12288 Mar 26 12:18 'pins_Sarah Wayne Callies'
drwxr-xr-x 2 root root 12288 Mar 26 12:18 'pins_Scarlett Johansson'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_sean pertwee'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Sebastian Stan'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_selena gomez'
drwxr-xr-x 2 root root  4096 Mar 26 12:18  pins_shakira
drwxr-xr-x 2 root root 12288 Mar 26 12:18 'pins_Sophie Turner'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Stephen Amell'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Sundar Pichai'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_tati gabrielle'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_taylor swift'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Thomas Middleditch'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Tom Cavanagh'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_tom holland face'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Ursula Corbero'
drwxr-xr-x 2 root root  4096 Mar 26 12:18 'pins_Wentworth Miller'
drwxr-xr-x 2 root root 12288 Mar 26 12:18 'pins_Willa Holland'
drwxr-xr-x 2 root root 12288 Mar 26 12:18 'pins_William Fichtner'
drwxr-xr-x 2 root root  4096 Mar 26 12:18  pins_zendaya
In [ ]:
!ls -l pins/PINS | wc -l
101
In [ ]:
!ls -l PINS | wc -l
101
In [ ]:
!ls -l PINS/
total 568
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Aaron Paul'
drwxr-xr-x 2 root root 12288 Mar 26 04:33 'pins_alexandra daddario'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Alvaro Morte'
drwxr-xr-x 2 root root 12288 Mar 26 04:33 'pins_alycia debnam carey face'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Amanda Crew'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Amaury Nolasco'
drwxr-xr-x 2 root root 12288 Mar 26 04:33 'pins_amber heard face'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Anna Gunn'
drwxr-xr-x 2 root root 12288 Mar 26 04:33 'pins_anne hathaway'
drwxr-xr-x 2 root root 12288 Mar 26 04:33 'pins_barbara palvin face'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_bellamy blake face'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Benedict Cumberbatch'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Betsy Brandt'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_bill gates'
drwxr-xr-x 2 root root 12288 Mar 26 04:33 'pins_Brenton Thwaites'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_brie larson'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Brit Marling'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Bryan Cranston'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Caity Lotz'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Cameron Monaghan'
drwxr-xr-x 2 root root 12288 Mar 26 04:33 'pins_chadwick boseman face'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Chance Perdomo'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Chris Evans'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Chris Pratt'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Cobie Smulders'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Danielle Panabaker'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Dave Franco'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_david mazouz'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Dominic Purcell'
drwxr-xr-x 2 root root  4096 Mar 26 04:33  pins_drake
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_dua lipa face'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Dwayne Johnson'
drwxr-xr-x 2 root root 12288 Mar 26 04:33 'pins_elizabeth olsen face'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_eliza taylor'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_elon musk'
drwxr-xr-x 2 root root 12288 Mar 26 04:33 'pins_Emilia Clarke'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Emily Bett Rickards'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Emma Stone'
drwxr-xr-x 2 root root 12288 Mar 26 04:33 'pins_emma watson face'
drwxr-xr-x 2 root root 12288 Mar 26 04:33 'pins_gal gadot face'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_grant gustin face'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Gwyneth Paltrow'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Henry Cavil'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_jason isaacs'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Jason Momoa'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_jeff bezos'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Jeremy Renner'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Jesse Eisenberg'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Jim Parsons'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Jon Bernthal'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Josh Radnor'
drwxr-xr-x 2 root root 12288 Mar 26 04:33 'pins_kiernan shipka '
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Kit Harington'
drwxr-xr-x 2 root root 12288 Mar 26 04:33 'pins_kristen stewart face'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Krysten Ritter'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Kumail Nanjiani'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_lindsey morgan face'
drwxr-xr-x 2 root root 12288 Mar 26 04:33 'pins_Maisie Williams'
drwxr-xr-x 2 root root 12288 Mar 26 04:33 'pins_margot robbie face'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_maria pedraza'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Mark Ruffalo'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_mark zuckerberg'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Martin Starr'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Melissa benoit'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_miguel herran'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Mike Colter'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_millie bobby brown'
drwxr-xr-x 2 root root 12288 Mar 26 04:33 'pins_Morena Baccarin'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Morgan Freeman'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Natalie Portman '
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Neil Patrick Harris'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Paul Rudd'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Pedro Alonso'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Peter Dinklage'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Rami Melek'
drwxr-xr-x 2 root root  4096 Mar 26 04:33  pins_rihanna
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_RJ Mitte'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_robert downey jr face'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Robert Knepper'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Robin Taylor'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Ryan Reynolds'
drwxr-xr-x 2 root root 12288 Mar 26 04:33 'pins_Sarah Wayne Callies'
drwxr-xr-x 2 root root 12288 Mar 26 04:33 'pins_Scarlett Johansson'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_sean pertwee'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Sebastian Stan'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_selena gomez'
drwxr-xr-x 2 root root  4096 Mar 26 04:33  pins_shakira
drwxr-xr-x 2 root root 12288 Mar 26 04:33 'pins_Sophie Turner'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Stephen Amell'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Sundar Pichai'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_tati gabrielle'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_taylor swift'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Thomas Middleditch'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Tom Cavanagh'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_tom holland face'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Ursula Corbero'
drwxr-xr-x 2 root root  4096 Mar 26 04:33 'pins_Wentworth Miller'
drwxr-xr-x 2 root root 12288 Mar 26 04:33 'pins_Willa Holland'
drwxr-xr-x 2 root root 12288 Mar 26 04:33 'pins_William Fichtner'
drwxr-xr-x 2 root root  4096 Mar 26 04:33  pins_zendaya
In [ ]:
!ls -l 'PINS/pins_Jim Parsons'
total 2588
-rw-r--r-- 1 root root 19071 Oct 21  2019 'Jim Parsons0.jpg'
-rw-r--r-- 1 root root 20054 Oct 21  2019 'Jim Parsons100.jpg'
-rw-r--r-- 1 root root 20086 Oct 21  2019 'Jim Parsons101.jpg'
-rw-r--r-- 1 root root 25099 Oct 21  2019 'Jim Parsons102.jpg'
-rw-r--r-- 1 root root 23095 Oct 21  2019 'Jim Parsons103.jpg'
-rw-r--r-- 1 root root 30489 Oct 21  2019 'Jim Parsons104.jpg'
-rw-r--r-- 1 root root 23304 Oct 21  2019 'Jim Parsons105.jpg'
-rw-r--r-- 1 root root 18134 Oct 21  2019 'Jim Parsons106.jpg'
-rw-r--r-- 1 root root 21270 Oct 21  2019 'Jim Parsons107.jpg'
-rw-r--r-- 1 root root 25884 Oct 21  2019 'Jim Parsons108.jpg'
-rw-r--r-- 1 root root 18024 Oct 21  2019 'Jim Parsons10.jpg'
-rw-r--r-- 1 root root 20894 Oct 21  2019 'Jim Parsons11.jpg'
-rw-r--r-- 1 root root 23343 Oct 21  2019 'Jim Parsons12.jpg'
-rw-r--r-- 1 root root 23232 Oct 21  2019 'Jim Parsons13.jpg'
-rw-r--r-- 1 root root 17757 Oct 21  2019 'Jim Parsons14.jpg'
-rw-r--r-- 1 root root 31904 Oct 21  2019 'Jim Parsons15.jpg'
-rw-r--r-- 1 root root 20781 Oct 21  2019 'Jim Parsons16.jpg'
-rw-r--r-- 1 root root 22075 Oct 21  2019 'Jim Parsons17.jpg'
-rw-r--r-- 1 root root 22622 Oct 21  2019 'Jim Parsons18.jpg'
-rw-r--r-- 1 root root 20785 Oct 21  2019 'Jim Parsons19.jpg'
-rw-r--r-- 1 root root 25650 Oct 21  2019 'Jim Parsons1.jpg'
-rw-r--r-- 1 root root 25453 Oct 21  2019 'Jim Parsons20.jpg'
-rw-r--r-- 1 root root 20984 Oct 21  2019 'Jim Parsons21.jpg'
-rw-r--r-- 1 root root 26703 Oct 21  2019 'Jim Parsons22.jpg'
-rw-r--r-- 1 root root 20713 Oct 21  2019 'Jim Parsons23.jpg'
-rw-r--r-- 1 root root 21984 Oct 21  2019 'Jim Parsons24.jpg'
-rw-r--r-- 1 root root 21613 Oct 21  2019 'Jim Parsons25.jpg'
-rw-r--r-- 1 root root 20298 Oct 21  2019 'Jim Parsons26.jpg'
-rw-r--r-- 1 root root 28378 Oct 21  2019 'Jim Parsons27.jpg'
-rw-r--r-- 1 root root 22557 Oct 21  2019 'Jim Parsons28.jpg'
-rw-r--r-- 1 root root 21554 Oct 21  2019 'Jim Parsons29.jpg'
-rw-r--r-- 1 root root 20663 Oct 21  2019 'Jim Parsons2.jpg'
-rw-r--r-- 1 root root 26778 Oct 21  2019 'Jim Parsons30.jpg'
-rw-r--r-- 1 root root 18849 Oct 21  2019 'Jim Parsons31.jpg'
-rw-r--r-- 1 root root 14891 Oct 21  2019 'Jim Parsons32.jpg'
-rw-r--r-- 1 root root 23546 Oct 21  2019 'Jim Parsons33.jpg'
-rw-r--r-- 1 root root 27090 Oct 21  2019 'Jim Parsons34.jpg'
-rw-r--r-- 1 root root 24901 Oct 21  2019 'Jim Parsons35.jpg'
-rw-r--r-- 1 root root 25491 Oct 21  2019 'Jim Parsons36.jpg'
-rw-r--r-- 1 root root 22394 Oct 21  2019 'Jim Parsons37.jpg'
-rw-r--r-- 1 root root 19787 Oct 21  2019 'Jim Parsons38.jpg'
-rw-r--r-- 1 root root 24423 Oct 21  2019 'Jim Parsons39.jpg'
-rw-r--r-- 1 root root 21869 Oct 21  2019 'Jim Parsons3.jpg'
-rw-r--r-- 1 root root 20924 Oct 21  2019 'Jim Parsons40.jpg'
-rw-r--r-- 1 root root 15642 Oct 21  2019 'Jim Parsons41.jpg'
-rw-r--r-- 1 root root 27039 Oct 21  2019 'Jim Parsons42.jpg'
-rw-r--r-- 1 root root 21191 Oct 21  2019 'Jim Parsons43.jpg'
-rw-r--r-- 1 root root 23020 Oct 21  2019 'Jim Parsons44.jpg'
-rw-r--r-- 1 root root 26654 Oct 21  2019 'Jim Parsons45.jpg'
-rw-r--r-- 1 root root 17587 Oct 21  2019 'Jim Parsons46.jpg'
-rw-r--r-- 1 root root 23021 Oct 21  2019 'Jim Parsons47.jpg'
-rw-r--r-- 1 root root 18534 Oct 21  2019 'Jim Parsons48.jpg'
-rw-r--r-- 1 root root 23630 Oct 21  2019 'Jim Parsons49.jpg'
-rw-r--r-- 1 root root 30022 Oct 21  2019 'Jim Parsons4.jpg'
-rw-r--r-- 1 root root 22131 Oct 21  2019 'Jim Parsons50.jpg'
-rw-r--r-- 1 root root 24327 Oct 21  2019 'Jim Parsons51.jpg'
-rw-r--r-- 1 root root 25304 Oct 21  2019 'Jim Parsons52.jpg'
-rw-r--r-- 1 root root 17704 Oct 21  2019 'Jim Parsons53.jpg'
-rw-r--r-- 1 root root 20721 Oct 21  2019 'Jim Parsons54.jpg'
-rw-r--r-- 1 root root 22135 Oct 21  2019 'Jim Parsons55.jpg'
-rw-r--r-- 1 root root 22573 Oct 21  2019 'Jim Parsons56.jpg'
-rw-r--r-- 1 root root 20685 Oct 21  2019 'Jim Parsons57.jpg'
-rw-r--r-- 1 root root 25599 Oct 21  2019 'Jim Parsons58.jpg'
-rw-r--r-- 1 root root 18232 Oct 21  2019 'Jim Parsons59.jpg'
-rw-r--r-- 1 root root 20567 Oct 21  2019 'Jim Parsons5.jpg'
-rw-r--r-- 1 root root 16075 Oct 21  2019 'Jim Parsons60.jpg'
-rw-r--r-- 1 root root 21760 Oct 21  2019 'Jim Parsons61.jpg'
-rw-r--r-- 1 root root 21210 Oct 21  2019 'Jim Parsons62.jpg'
-rw-r--r-- 1 root root 20545 Oct 21  2019 'Jim Parsons63.jpg'
-rw-r--r-- 1 root root 21364 Oct 21  2019 'Jim Parsons64.jpg'
-rw-r--r-- 1 root root 19660 Oct 21  2019 'Jim Parsons65.jpg'
-rw-r--r-- 1 root root 14364 Oct 21  2019 'Jim Parsons66.jpg'
-rw-r--r-- 1 root root 21963 Oct 21  2019 'Jim Parsons67.jpg'
-rw-r--r-- 1 root root 21998 Oct 21  2019 'Jim Parsons68.jpg'
-rw-r--r-- 1 root root 28491 Oct 21  2019 'Jim Parsons69.jpg'
-rw-r--r-- 1 root root 15701 Oct 21  2019 'Jim Parsons6.jpg'
-rw-r--r-- 1 root root 18556 Oct 21  2019 'Jim Parsons70.jpg'
-rw-r--r-- 1 root root 21741 Oct 21  2019 'Jim Parsons71.jpg'
-rw-r--r-- 1 root root 27067 Oct 21  2019 'Jim Parsons72.jpg'
-rw-r--r-- 1 root root 26830 Oct 21  2019 'Jim Parsons73.jpg'
-rw-r--r-- 1 root root 24390 Oct 21  2019 'Jim Parsons74.jpg'
-rw-r--r-- 1 root root 30907 Oct 21  2019 'Jim Parsons75.jpg'
-rw-r--r-- 1 root root 25041 Oct 21  2019 'Jim Parsons76.jpg'
-rw-r--r-- 1 root root 18963 Oct 21  2019 'Jim Parsons77.jpg'
-rw-r--r-- 1 root root 21263 Oct 21  2019 'Jim Parsons78.jpg'
-rw-r--r-- 1 root root 18224 Oct 21  2019 'Jim Parsons79.jpg'
-rw-r--r-- 1 root root 24226 Oct 21  2019 'Jim Parsons7.jpg'
-rw-r--r-- 1 root root 20170 Oct 21  2019 'Jim Parsons80.jpg'
-rw-r--r-- 1 root root 22601 Oct 21  2019 'Jim Parsons81.jpg'
-rw-r--r-- 1 root root 24182 Oct 21  2019 'Jim Parsons82.jpg'
-rw-r--r-- 1 root root 29867 Oct 21  2019 'Jim Parsons83.jpg'
-rw-r--r-- 1 root root 26659 Oct 21  2019 'Jim Parsons84.jpg'
-rw-r--r-- 1 root root 23254 Oct 21  2019 'Jim Parsons85.jpg'
-rw-r--r-- 1 root root 16779 Oct 21  2019 'Jim Parsons86.jpg'
-rw-r--r-- 1 root root 25684 Oct 21  2019 'Jim Parsons87.jpg'
-rw-r--r-- 1 root root 19725 Oct 21  2019 'Jim Parsons88.jpg'
-rw-r--r-- 1 root root 21240 Oct 21  2019 'Jim Parsons89.jpg'
-rw-r--r-- 1 root root 21293 Oct 21  2019 'Jim Parsons8.jpg'
-rw-r--r-- 1 root root 20049 Oct 21  2019 'Jim Parsons90.jpg'
-rw-r--r-- 1 root root 24256 Oct 21  2019 'Jim Parsons91.jpg'
-rw-r--r-- 1 root root 19360 Oct 21  2019 'Jim Parsons92.jpg'
-rw-r--r-- 1 root root 17504 Oct 21  2019 'Jim Parsons93.jpg'
-rw-r--r-- 1 root root 16144 Oct 21  2019 'Jim Parsons94.jpg'
-rw-r--r-- 1 root root 22824 Oct 21  2019 'Jim Parsons95.jpg'
-rw-r--r-- 1 root root 24020 Oct 21  2019 'Jim Parsons96.jpg'
-rw-r--r-- 1 root root 20180 Oct 21  2019 'Jim Parsons97.jpg'
-rw-r--r-- 1 root root 19288 Oct 21  2019 'Jim Parsons98.jpg'
-rw-r--r-- 1 root root 26660 Oct 21  2019 'Jim Parsons99.jpg'
-rw-r--r-- 1 root root 17601 Oct 21  2019 'Jim Parsons9.jpg'
In [ ]:
BASE_PATH = '/content/PINS'
In [ ]:
# initialise the dataframe to hold the metadata
# number of images is redundant and can be omitted
metadata_df = pd.DataFrame(columns=['PERSON','PERSON_BASE_PATH','FILES', 'NUMBER_OF_IMAGES'])

for dirs in os.listdir(BASE_PATH): # list all directories or persons

  if 'PINS_' in dirs.upper(): # check case-insensitive for pinterest
  
    person = dirs[5:].replace(' ','_').upper() # replace space with _
    person_base_path = os.path.join(BASE_PATH,dirs) # get base directory path of each person
    files = [] # initialise a list to hold all corresponding files

    for img_path in os.listdir(os.path.join(BASE_PATH,dirs)):
      files.append(img_path) # get each image path
    number_of_images = len(files) # calculate number of images

    # append to data-frame
    metadata_df.loc[metadata_df.shape[0]] = [person , person_base_path , files , number_of_images]
In [ ]:
# verify top records of data-frame
metadata_df.head()
Out[ ]:
PERSON PERSON_BASE_PATH FILES NUMBER_OF_IMAGES
0 EMMA_WATSON_FACE /content/PINS/pins_emma watson face [emma watson face1.jpg, emma watson face4.jpg,... 163
1 GRANT_GUSTIN_FACE /content/PINS/pins_grant gustin face [grant gustin face66.jpg, grant gustin face73.... 122
2 BRIT_MARLING /content/PINS/pins_Brit Marling [Brit Marling64_541.jpg, Brit Marling78_453.jp... 122
3 BETSY_BRANDT /content/PINS/pins_Betsy Brandt [Betsy Brandt116_429.jpg, Betsy Brandt122_416.... 68
4 TATI_GABRIELLE /content/PINS/pins_tati gabrielle [tati gabrielle33.jpg, tati gabrielle18.jpg, t... 65
In [ ]:
# verify shape of data-frame
metadata_df.shape
Out[ ]:
(100, 4)
In [ ]:
# checking if all the persons are unique
metadata_df['PERSON'].nunique()
Out[ ]:
100
In [ ]:
# verifying total number of images
total_images = metadata_df['NUMBER_OF_IMAGES'].sum()
total_images
Out[ ]:
10770

Observations:

  • There are total of 100 persons.
  • For each person there are varying number of images.
  • Total number of images are 10770.

Visualisation

We will try to visualise randomly 10 images for 10 persons.

In [ ]:
# get 10 random index for persons
person_idices = np.random.choice(metadata_df.shape[0],10)

# initialise figure parameters
fig = plt.figure(1, (100, 100))
grid = ImageGrid(fig, 111, nrows_ncols=(10, 10), axes_pad=0.1)
counter = 0

for person_idx in person_idices:
  # get a record from the dataframe
  record = metadata_df.iloc[person_idx]
  person = record['PERSON']
  person_base_path = record['PERSON_BASE_PATH']
  number_of_images = record['NUMBER_OF_IMAGES'] - 1
  files = record['FILES']

  # get randomly 10 indices of images for a person
  image_indices = np.random.choice(number_of_images, 10)
  display_title = True # display title or image

  # display images for a person
  for img_idx in image_indices:
    ax = grid[counter]
    if display_title:
        ax.text(1700, 105, person, verticalalignment='center',fontsize=60)
        display_title = False
    img_path = person_base_path + '/' + files[img_idx]
    img = cv2.imread(img_path)
    img = cv2.resize(img, (150, 150))
    ax.axis("off")
    ax.imshow(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))
    counter += 1

plt.show()
Output hidden; open in https://colab.research.google.com to view.

Load a pre-trained model and it's weights

In [ ]:
# define model params
IMG_SIZE = 224
In [ ]:
# load a model for assigning the pre-trained weights
def vgg_face(img_size=224):	
    model = tf.keras.models.Sequential() # initialise the model

    model.add(tf.keras.layers.ZeroPadding2D((1,1),input_shape=(img_size,img_size, 3)))
    model.add(tf.keras.layers.Convolution2D(64, (3, 3), activation='relu'))
    model.add(tf.keras.layers.ZeroPadding2D((1,1)))
    model.add(tf.keras.layers.Convolution2D(64, (3, 3), activation='relu'))
    model.add(tf.keras.layers.MaxPooling2D((2,2), strides=(2,2)))
    
    model.add(tf.keras.layers.ZeroPadding2D((1,1)))
    model.add(tf.keras.layers.Convolution2D(128, (3, 3), activation='relu'))
    model.add(tf.keras.layers.ZeroPadding2D((1,1)))
    model.add(tf.keras.layers.Convolution2D(128, (3, 3), activation='relu'))
    model.add(tf.keras.layers.MaxPooling2D((2,2), strides=(2,2)))
    
    model.add(tf.keras.layers.ZeroPadding2D((1,1)))
    model.add(tf.keras.layers.Convolution2D(256, (3, 3), activation='relu'))
    model.add(tf.keras.layers.ZeroPadding2D((1,1)))
    model.add(tf.keras.layers.Convolution2D(256, (3, 3), activation='relu'))
    model.add(tf.keras.layers.ZeroPadding2D((1,1)))
    model.add(tf.keras.layers.Convolution2D(256, (3, 3), activation='relu'))
    model.add(tf.keras.layers.MaxPooling2D((2,2), strides=(2,2)))
    
    model.add(tf.keras.layers.ZeroPadding2D((1,1)))
    model.add(tf.keras.layers.Convolution2D(512, (3, 3), activation='relu'))
    model.add(tf.keras.layers.ZeroPadding2D((1,1)))
    model.add(tf.keras.layers.Convolution2D(512, (3, 3), activation='relu'))
    model.add(tf.keras.layers.ZeroPadding2D((1,1)))
    model.add(tf.keras.layers.Convolution2D(512, (3, 3), activation='relu'))
    model.add(tf.keras.layers.MaxPooling2D((2,2), strides=(2,2)))
    
    model.add(tf.keras.layers.ZeroPadding2D((1,1)))
    model.add(tf.keras.layers.Convolution2D(512, (3, 3), activation='relu'))
    model.add(tf.keras.layers.ZeroPadding2D((1,1)))
    model.add(tf.keras.layers.Convolution2D(512, (3, 3), activation='relu'))
    model.add(tf.keras.layers.ZeroPadding2D((1,1)))
    model.add(tf.keras.layers.Convolution2D(512, (3, 3), activation='relu'))
    model.add(tf.keras.layers.MaxPooling2D((2,2), strides=(2,2)))
    
    model.add(tf.keras.layers.Convolution2D(4096, (7, 7), activation='relu'))
    model.add(tf.keras.layers.Dropout(0.5))
    model.add(tf.keras.layers.Convolution2D(4096, (1, 1), activation='relu'))
    model.add(tf.keras.layers.Dropout(0.5))
    model.add(tf.keras.layers.Convolution2D(2622, (1, 1)))
    model.add(tf.keras.layers.Flatten())
    model.add(tf.keras.layers.Activation('softmax'))
    return model
In [ ]:
# copy weights to current directory
!cp '/content/drive/MyDrive/Colab Notebooks/vgg_face_weights.h5' .
In [ ]:
# verify the contents
!ls -l
total 1034052
-rw-------   1 root root 478756252 Mar 26 12:18 Aligned+Face+Dataset+from+Pinterest+-+CV+project+1.zip
drwx------   5 root root      4096 Mar 26 12:16 drive
drwxr-xr-x   3 root root      4096 Mar 26 12:18 pins
drwxr-xr-x 102 root root      4096 Mar 26 12:18 PINS
drwxr-xr-x   1 root root      4096 Mar 23 14:22 sample_data
-rw-------   1 root root 580085408 Mar 26 12:19 vgg_face_weights.h5
In [ ]:
# build a placeholder model for loading pre-trained weights
model = vgg_face(IMG_SIZE)
In [ ]:
# load weights to the placeholder model
model.load_weights('/content/vgg_face_weights.h5')
In [ ]:
# getting input layer
model.layers[0].input
Out[ ]:
<KerasTensor: shape=(None, 224, 224, 3) dtype=float32 (created by layer 'zero_padding2d_input')>
In [ ]:
# getting output layer
model.layers[-1].output
Out[ ]:
<KerasTensor: shape=(None, 2622) dtype=float32 (created by layer 'activation')>
In [ ]:
# getting embedding layer
model.layers[-2].output
Out[ ]:
<KerasTensor: shape=(None, 2622) dtype=float32 (created by layer 'flatten')>
In [ ]:
# generating a VGG Face descriptor from the model
vgg_face_descriptor = tf.keras.models.Model(inputs=model.layers[0].input, outputs=model.layers[-2].output)
In [ ]:
# getting the dimensions of each emebedding vector
embedding_dimensions = vgg_face_descriptor.output.shape[1]
embedding_dimensions
Out[ ]:
2622

Observations:

  • The model expects an input tensor of 224x224x3, thus some images would require re-sizing and dimension expansion.
  • The output has to be extracted before the classifier layer so as to get the emebedding vectors of an image, which can be used for distance/similarity comparison.

Generate Embedding Vectors

In [ ]:
# generate embedding vectors for each image

embedding_vectors =  np.zeros((total_images,embedding_dimensions)) # initialise array to hold embedding vectors
embedding_counter = 0

classes = []
for idx in metadata_df.index: # for each record in the data-frame

  each_person = metadata_df.iloc[idx]
  each_person_name = each_person['PERSON']
  each_person_base_path = each_person['PERSON_BASE_PATH']
  each_person_images = each_person['FILES']

  for each_img_idx in range(0,each_person['NUMBER_OF_IMAGES']): # for each image of a person

    # Read - resize - normalize the image
    img_path = each_person_base_path + '/' + each_person_images[each_img_idx]
    img = tf.keras.preprocessing.image.load_img(img_path)
    img_arr = tf.keras.preprocessing.image.img_to_array(img.resize([IMG_SIZE,IMG_SIZE]))

    # Normalising pixel values from [0-255] to [0-1]: scale RGB values to interval [0,1]
    img_arr = (img_arr / 255.).astype(np.float32)

    # get the embedding vector for each image tensor
    embedding_vectors[embedding_counter] = vgg_face_descriptor.predict(np.expand_dims(img_arr,axis=0))[0]
    embedding_counter +=1

    # get labels 
    classes.append(each_person_name)
In [ ]:
# verifying the shape of the embedding-vectors
embedding_vectors.shape
Out[ ]:
(10770, 2622)
In [ ]:
# verifying few records of the embedding-vectors
embedding_vectors[:10]
Out[ ]:
array([[ 0.0050472 , -0.00430801, -0.0080788 , ..., -0.01274249,
         0.01267068,  0.01951582],
       [ 0.01578406, -0.0115249 ,  0.01688461, ..., -0.02433562,
         0.00350092,  0.01615274],
       [-0.00285404, -0.01575482, -0.01084395, ..., -0.00994424,
        -0.00640306,  0.00934648],
       ...,
       [ 0.00644071,  0.0031547 , -0.00069348, ..., -0.01011371,
         0.00060886,  0.02488079],
       [-0.00235213, -0.00317539,  0.00408626, ..., -0.00564185,
        -0.00172605,  0.02105103],
       [-0.00094291, -0.00751995, -0.00854421, ..., -0.02040366,
        -0.00606529,  0.00617395]])
In [ ]:
np.array(classes).shape
Out[ ]:
(10770,)

Distance Metric between Embeddings

For calculation of similarity or dissimilarity between emebedding vectors, we can use various metrics like:

  • Distance Metrics:
    • L1 distance
    • L2 distance
  • Similarity Metrics:
    • Cosine Similarity
    • Jaccard Similarity

Here, we will make use of L2 distance.

In [ ]:
def l2_distance_metric(emb1, emb2):
  '''
  Calculate Sum of squares of differences
  '''
  return np.sum(np.square(emb1 - emb2))
In [ ]:
def get_normalized_img_arr(img_path):
  '''
  Gets a normalized image array from the image path
  '''
  img = tf.keras.preprocessing.image.load_img(img_path)
  img_arr = tf.keras.preprocessing.image.img_to_array(img.resize([IMG_SIZE,IMG_SIZE]))

  # Normalising pixel values from [0-255] to [0-1]: scale RGB values to interval [0,1]
  img_arr = (img_arr / 255.).astype(np.float32)

  return img_arr
In [ ]:
def get_img_arr_name(idx):
  '''
  Gives a pair of image array and name
  '''
  person_record = metadata_df.iloc[idx]
  person_name = person_record['PERSON']
  person_base_path = person_record['PERSON_BASE_PATH']
  person_files = person_record['FILES']
  person_number_of_images = person_record['NUMBER_OF_IMAGES']

  random_images = np.random.randint(0,person_number_of_images,2)
  # Read - resize - normalize the image
  img_path = person_base_path + '/' + person_files[random_images[0]]
  img_arr1 = get_normalized_img_arr(img_path)

  img_path = person_base_path + '/' + person_files[random_images[1]]
  img_arr2 = get_normalized_img_arr(img_path)

  return person_name,img_arr1, img_arr2
In [ ]:
def get_positive_negative_pairs():
  '''
  Gives a positive pair of embeddings and negative pair of embeddings
  '''
  random_pair_indices = np.random.randint(0,metadata_df.shape[0],2)

  # first person details
  person1, img_arr11, img_arr12 = get_img_arr_name(random_pair_indices[0])

  # second person details
  person2, img_arr21, img_arr22 = get_img_arr_name(random_pair_indices[1])

  # get embeddings
  emb11 = vgg_face_descriptor.predict(np.expand_dims(img_arr11,axis=0))[0]
  emb12 = vgg_face_descriptor.predict(np.expand_dims(img_arr12,axis=0))[0]
  emb21 = vgg_face_descriptor.predict(np.expand_dims(img_arr21,axis=0))[0]
  emb22 = vgg_face_descriptor.predict(np.expand_dims(img_arr22,axis=0))[0]

  positive_pair = ((person1,person1),(emb11,emb12),(img_arr11, img_arr12))
  negative_pair = ((person1,person2),(emb11,emb21),(img_arr11, img_arr21))

  return positive_pair,negative_pair
In [ ]:
# display embeddings and corresponding images
def show_pair(pair):
  # initialising subplots
  figure, ax = plt.subplots(nrows=1, ncols=2,constrained_layout=True)

  # setting figure parameters
  figure.set_figheight(15)
  figure.set_figwidth(15)

  figure.suptitle(f'Distance = {l2_distance_metric(pair[1][0], pair[1][1]):.4f}', fontsize=16)

  # setting images
  ax[0].imshow(pair[2][0])
  ax[0].set_title(pair[0][0])
  ax[0].axis('off')
  ax[1].imshow(pair[2][1])
  ax[1].set_title(pair[0][1])
  ax[1].axis('off')

  plt.tight_layout()
  plt.show()
In [ ]:
positive_pair,negative_pair = get_positive_negative_pairs()
In [ ]:
positive_pair
Out[ ]:
(('JEFF_BEZOS', 'JEFF_BEZOS'),
 (array([ 0.00210719, -0.00891261,  0.01853747, ..., -0.00938883,
          0.01012283,  0.00715792], dtype=float32),
  array([ 0.01421697, -0.02232722,  0.01497093, ..., -0.00877767,
          0.01047427, -0.00585749], dtype=float32)),
 (array([[[0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.],
          ...,
          [0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.]],
  
         [[0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.],
          ...,
          [0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.]],
  
         [[0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.],
          ...,
          [0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.]],
  
         ...,
  
         [[0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.],
          ...,
          [0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.]],
  
         [[0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.],
          ...,
          [0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.]],
  
         [[0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.],
          ...,
          [0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.]]], dtype=float32),
  array([[[0.73333335, 0.7372549 , 0.7058824 ],
          [0.74509805, 0.7411765 , 0.70980394],
          [0.7490196 , 0.7490196 , 0.7176471 ],
          ...,
          [0.05098039, 0.00392157, 0.00392157],
          [0.04313726, 0.00392157, 0.        ],
          [0.03921569, 0.        , 0.        ]],
  
         [[0.7372549 , 0.7372549 , 0.7058824 ],
          [0.7411765 , 0.7411765 , 0.70980394],
          [0.7490196 , 0.7490196 , 0.7176471 ],
          ...,
          [0.05098039, 0.00392157, 0.00392157],
          [0.03921569, 0.        , 0.        ],
          [0.03921569, 0.        , 0.        ]],
  
         [[0.73333335, 0.73333335, 0.7019608 ],
          [0.74509805, 0.74509805, 0.7137255 ],
          [0.7490196 , 0.7490196 , 0.7176471 ],
          ...,
          [0.04705882, 0.        , 0.        ],
          [0.03921569, 0.00392157, 0.        ],
          [0.03921569, 0.        , 0.        ]],
  
         ...,
  
         [[0.07843138, 0.0627451 , 0.05098039],
          [0.07843138, 0.0627451 , 0.05098039],
          [0.07058824, 0.05490196, 0.04313726],
          ...,
          [0.11372549, 0.08235294, 0.1254902 ],
          [0.10196079, 0.06666667, 0.10980392],
          [0.08627451, 0.04313726, 0.08235294]],
  
         [[0.07843138, 0.0627451 , 0.05098039],
          [0.07450981, 0.05882353, 0.04705882],
          [0.06666667, 0.05098039, 0.03921569],
          ...,
          [0.11764706, 0.08627451, 0.12941177],
          [0.10588235, 0.07058824, 0.11372549],
          [0.09019608, 0.05098039, 0.09411765]],
  
         [[0.07843138, 0.0627451 , 0.05098039],
          [0.07450981, 0.05882353, 0.04705882],
          [0.06666667, 0.05098039, 0.03921569],
          ...,
          [0.11372549, 0.08235294, 0.12941177],
          [0.09803922, 0.06666667, 0.10980392],
          [0.08235294, 0.04705882, 0.09019608]]], dtype=float32)))
In [ ]:
show_pair(positive_pair)
In [ ]:
negative_pair
Out[ ]:
(('JEFF_BEZOS', 'BRIT_MARLING'),
 (array([ 0.00210719, -0.00891261,  0.01853747, ..., -0.00938883,
          0.01012283,  0.00715792], dtype=float32),
  array([ 0.0125706 ,  0.0097119 , -0.00091843, ..., -0.03644164,
         -0.0009713 ,  0.0062989 ], dtype=float32)),
 (array([[[0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.],
          ...,
          [0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.]],
  
         [[0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.],
          ...,
          [0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.]],
  
         [[0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.],
          ...,
          [0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.]],
  
         ...,
  
         [[0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.],
          ...,
          [0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.]],
  
         [[0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.],
          ...,
          [0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.]],
  
         [[0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.],
          ...,
          [0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.]]], dtype=float32), array([[[0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.],
          ...,
          [0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.]],
  
         [[0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.],
          ...,
          [0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.]],
  
         [[0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.],
          ...,
          [0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.]],
  
         ...,
  
         [[0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.],
          ...,
          [0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.]],
  
         [[0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.],
          ...,
          [0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.]],
  
         [[0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.],
          ...,
          [0., 0., 0.],
          [0., 0., 0.],
          [0., 0., 0.]]], dtype=float32)))
In [ ]:
show_pair(negative_pair)

Observations:

  • The L2 distance between the embeddings of a positive-pair(same images) is low, indicating high similarity.
  • The L2 distance between the embeddings of a negative-pair(different images) is high, indicating high dissimilarity.

Split in train and Validation sets

In [ ]:
# every 9th example goes in test data and rest go in train data
train_idx = np.arange(embedding_vectors.shape[0]) % 9 != 0     
test_idx = np.arange(embedding_vectors.shape[0]) % 9 == 0

# one half as train examples of 10 identities
X_train = embedding_vectors[train_idx]

# another half as test examples of 10 identities
X_test = embedding_vectors[test_idx]

classes = np.array(classes)
#train labels
y_train = classes[train_idx]

#test labels
y_test = classes[test_idx]

print(f'X_train shape : ({X_train.shape[0]},{X_train.shape[1]})')
print(f'y_train shape : ({y_train.shape[0]},)')
print(f'X_test shape : ({X_test.shape[0]},{ X_test.shape[1]})')
print(f'y_test shape : ({y_test.shape[0]},)')
X_train shape : (9573,2622)
y_train shape : (9573,)
X_test shape : (1197,2622)
y_test shape : (1197,)

Encode the labels

In [ ]:
le = LabelEncoder() # initialise the encoder

# encode the labels
y_train_encoded = le.fit_transform(y_train) 
y_test_encoded = le.transform(y_test)
In [ ]:
y_train_encoded
Out[ ]:
array([38, 38, 38, ...,  5,  5,  5])
In [ ]:
y_test_encoded
Out[ ]:
array([38, 38, 38, ...,  5,  5,  5])

Standardise feature values

For PCA it is important to have normalized/scaled feature values so that the components are not biased towards specific feature values

In [ ]:
scaler = StandardScaler() # initialise the scaler

# scale the feature values
X_train_std = scaler.fit_transform(X_train)
X_test_std = scaler.transform(X_test)

Dimensionality Reduction using PCA

As each of our embeddings have 2622 features, we can reduce the dimensions.

PCA is a way of linearly transforming the data such that most of the information in the data is contained within a smaller number of features called components.

In [ ]:
def evaluate_the_model(model,model_name,X_train,y_train,X_test,y_test):
  # predict on test data
  y_pred = model.predict(X_test)
  predictions[model_name] = y_pred

  train_accuracy = model.score(X_train,y_train)
  test_accuracy = model.score(X_test,y_test)
  accuracies[model_name] = [train_accuracy,test_accuracy]

  print(f'The accuracy score of model on train data is {round(train_accuracy,4)*100}%')
  print(f'The accuracy score of model on test data is {round(test_accuracy,4)*100}%')
  print()

  acc= metrics.accuracy_score(y_test, y_pred)
  print('Accuracy Score :','%0.2f' % acc)
  model_lists.append([model_name, acc * 100])
In [ ]:
# initialised a dictionary to hold accuracy of various models
# {model: [train_accuracy,test_accuracy]}
accuracies = {}
In [ ]:
# initialised a dictionary to hold predictions of various models
# {model: [predictions]}
predictions = {}
In [ ]:
model_lists = [] # initialise to hold various model parameters
In [ ]:
pca = PCA(n_components=500) # initialisize PCA with 500 components

# transform the scaled feature values
X_train_pca = pca.fit_transform(X_train_std)
X_test_pca = pca.transform(X_test_std)

Building a Support Vector Machine

In [ ]:
# hyper-parameters to search from
param_grid = {'C': [0.1,1,5], 'gamma': [0.01,0.001],'kernel': ['rbf', 'linear']}

# grid search classifier
svm_grid= GridSearchCV(svm.SVC(), param_grid, verbose = 2,cv=5, n_jobs = -1)
 
# # fit on samples
svm_grid.fit(X_train_pca, y_train_encoded)

# getting the best parameters
svm_grid.best_params_
Fitting 5 folds for each of 12 candidates, totalling 60 fits
Out[ ]:
{'C': 0.1, 'gamma': 0.01, 'kernel': 'linear'}
In [ ]:
# call to the function
evaluate_the_model(svm_grid,'SVM with Grid Search CV',X_train_pca,y_train_encoded,X_test_pca,y_test_encoded)
The accuracy score of model on train data is 99.98%
The accuracy score of model on test data is 96.41%

Accuracy Score : 0.96
In [ ]:
model_df = pd.DataFrame(model_lists, columns = ['Model', 'Accuracy Scores on Test'])
model_df
Out[ ]:
Model Accuracy Scores on Test
0 SVM with Grid Search CV 96.407686
In [ ]:
accuracy_df = pd.DataFrame(accuracies,index=['Train Accuracy','Test Accuracy'])
accuracy_df
Out[ ]:
SVM with Grid Search CV
Train Accuracy 0.999791
Test Accuracy 0.964077

Observations:

  • We obtained the best hyper-parameters of the SVM using GridSearchCV.
  • The model performs well on train and validation data.

Predicting using trained SVM

In [ ]:
!cp '/content/drive/MyDrive/Colab Notebooks/Part 2 - Test Image - Dwayne Johnson4.jpg' .
In [ ]:
!cp '/content/drive/MyDrive/Colab Notebooks/Part 2- Test Image - Benedict Cumberbatch9.jpg' .
In [ ]:
!ls -l
total 1034108
-rw-------   1 root root 478756252 Mar 26 12:18  Aligned+Face+Dataset+from+Pinterest+-+CV+project+1.zip
drwx------   5 root root      4096 Mar 26 12:16  drive
-rw-------   1 root root     30905 Mar 26 13:59 'Part 2- Test Image - Benedict Cumberbatch9.jpg'
-rw-------   1 root root     24417 Mar 26 13:59 'Part 2 - Test Image - Dwayne Johnson4.jpg'
drwxr-xr-x   3 root root      4096 Mar 26 12:18  pins
drwxr-xr-x 102 root root      4096 Mar 26 12:18  PINS
drwxr-xr-x   1 root root      4096 Mar 23 14:22  sample_data
-rw-------   1 root root 580085408 Mar 26 12:19  vgg_face_weights.h5
In [ ]:
def prediction_pipeline(img_path, actual_label):
  img = tf.keras.preprocessing.image.load_img(img_path)
  img_arr = tf.keras.preprocessing.image.img_to_array(img.resize([IMG_SIZE,IMG_SIZE]))

  # Normalising pixel values from [0-255] to [0-1]: scale RGB values to interval [0,1]
  img_arr = (img_arr / 255.).astype(np.float32)

  # get the embedding vector for each image tensor
  embedding_vector = vgg_face_descriptor.predict(np.expand_dims(img_arr,axis=0))[0]

  # scale the feature values
  embedding_vector_scaled = scaler.transform(np.expand_dims(embedding_vector,axis=0))

  # transform the scaled feature values
  embedding_vector_pca = pca.transform(embedding_vector_scaled)

  # make a prediction
  y_pred = svm_grid.predict(embedding_vector_pca)

  print(f'Actual Label: {actual_label.upper()}')
  print(f'Predicted Encoded Label: {y_pred[0]}')
  print(f'Predicted Label: {le.inverse_transform(y_pred)[0]}')
In [ ]:
prediction_pipeline('/content/Part 2- Test Image - Benedict Cumberbatch9.jpg','Benedict Cumberbatch')
Actual Label: BENEDICT CUMBERBATCH
Predicted Encoded Label: 11
Predicted Label: BENEDICT_CUMBERBATCH
In [ ]:
tf.keras.preprocessing.image.load_img('/content/Part 2- Test Image - Benedict Cumberbatch9.jpg')
Out[ ]:
In [ ]:
prediction_pipeline('/content/Part 2 - Test Image - Dwayne Johnson4.jpg','Dwayne Johnson')
Actual Label: DWAYNE JOHNSON
Predicted Encoded Label: 31
Predicted Label: DWAYNE_JOHNSON
In [ ]:
tf.keras.preprocessing.image.load_img('/content/Part 2 - Test Image - Dwayne Johnson4.jpg')
Out[ ]:
In [ ]:
%%shell
jupyter nbconvert --to html '/content/drive/MyDrive/Colab Notebooks/ADVANCED_COMPUTER_VISION_PROJECT.ipynb'
[NbConvertApp] Converting notebook /content/drive/MyDrive/Colab Notebooks/ADVANCED_COMPUTER_VISION_PROJECT.ipynb to html
[NbConvertApp] Writing 4454337 bytes to /content/drive/MyDrive/Colab Notebooks/ADVANCED_COMPUTER_VISION_PROJECT.html
Out[ ]: